2024-03-28T14:50:06.340 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T14:50:06.340 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --owner scheduled_pdonnell@teuthology --name pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984 --verbose -- /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984/orig.config.yaml 2024-03-28T14:50:06.392 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T14:50:06.483 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984 branch: wip-batrick-testing-20240327.230800 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: pdonnell@redhat.com first_in_suite: false job_id: '7627984' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240327.230800 ceph: cephfs: session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: false mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - Replacing daemon mds sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240327.230800 sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 7752 sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 sleep_before_teardown: 0 subset: 1/512 suite: fs suite_branch: wip-batrick-testing-20240327.230800 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0PFIEkLyGui6wXmHsGy8PaMcdiY15HyvBvZeH/tU5plzXHweD4TBjDdVh/wR+38K6IBCIsbrxOSnb2KhpgaI4= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPl/NaYLYsV2jUAwfErgcSHMcwl9tx+xvwOnB5XTNmoI3kMfLwxGPgh4IyYfUkr9ouQzNjpKWUKCBhM0MBiE2I= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-28_07:14:07 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T14:50:06.483 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa; will attempt to use it 2024-03-28T14:50:06.484 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks 2024-03-28T14:50:06.484 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T14:50:06.486 INFO:teuthology.task.internal:Checking packages... 2024-03-28T14:50:06.510 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15' 2024-03-28T14:50:06.510 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T14:50:06.510 INFO:teuthology.packaging:ref: None 2024-03-28T14:50:06.510 INFO:teuthology.packaging:tag: None 2024-03-28T14:50:06.510 INFO:teuthology.packaging:branch: wip-batrick-testing-20240327.230800 2024-03-28T14:50:06.511 INFO:teuthology.packaging:sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T14:50:06.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240327.230800 2024-03-28T14:50:06.650 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2499.ge0e48ba6 2024-03-28T14:50:06.652 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T14:50:06.659 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T14:50:06.660 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T14:50:06.701 INFO:teuthology.task.internal:Saving configuration 2024-03-28T14:50:06.721 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T14:50:06.728 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T14:50:06.755 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 14:44:49.320815', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0PFIEkLyGui6wXmHsGy8PaMcdiY15HyvBvZeH/tU5plzXHweD4TBjDdVh/wR+38K6IBCIsbrxOSnb2KhpgaI4='} 2024-03-28T14:50:06.781 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 14:44:49.322392', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPl/NaYLYsV2jUAwfErgcSHMcwl9tx+xvwOnB5XTNmoI3kMfLwxGPgh4IyYfUkr9ouQzNjpKWUKCBhM0MBiE2I='} 2024-03-28T14:50:06.781 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T14:50:06.788 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-28T14:50:06.789 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-28T14:50:06.789 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T14:50:06.885 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3d3b9e3160>, signals=[15]) 2024-03-28T14:50:06.886 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T14:50:06.893 INFO:teuthology.task.internal:Opening connections... 2024-03-28T14:50:06.893 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T14:50:06.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:50:06.985 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-03-28T14:50:06.986 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:50:07.078 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T14:50:07.086 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-03-28T14:50:07.105 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-03-28T14:50:07.105 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-03-28T14:50:07.161 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T14:50:07.162 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T14:50:07.163 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-03-28T14:50:07.187 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-03-28T14:50:07.205 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-03-28T14:50:07.205 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-03-28T14:50:07.262 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-03-28T14:50:07.262 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-03-28T14:50:07.262 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-03-28T14:50:07.262 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-03-28T14:50:07.262 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-03-28T14:50:07.262 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-03-28T14:50:07.262 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T14:50:07.262 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-03-28T14:50:07.262 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-03-28T14:50:07.262 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T14:50:07.263 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-03-28T14:50:07.263 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T14:50:07.263 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T14:50:07.263 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T14:50:07.263 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-03-28T14:50:07.278 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T14:50:07.290 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T14:50:07.310 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T14:50:07.310 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-03-28T14:50:07.313 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-03-28T14:50:07.330 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T14:50:07.338 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T14:50:07.338 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-03-28T14:50:07.370 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-03-28T14:50:07.465 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T14:50:07.523 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T14:50:07.545 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-28T14:50:07.546 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-28T14:50:07.546 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T14:50:07.546 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T14:50:07.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:50:07.563 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-03-28T14:50:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:50:07.564 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-03-28T14:50:07.622 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-361.el9.x86_64 2024-03-28T14:50:07.623 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-361.el9.x86_64 2024-03-28T14:50:07.623 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.14.0-361.el9.x86_64 2024-03-28T14:50:07.623 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum install -y kernel 2024-03-28T14:50:07.624 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.14.0-361.el9.x86_64 2024-03-28T14:50:07.624 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2024-03-28T14:50:10.089 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 8.2 MB/s | 6.7 MB 00:00 2024-03-28T14:50:10.095 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 6.8 MB/s | 6.7 MB 00:00 2024-03-28T14:50:11.539 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-28T14:50:11.812 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-28T14:50:13.813 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-03-28T14:50:14.769 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.0 MB 00:02 2024-03-28T14:50:17.472 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 19 MB 00:02 2024-03-28T14:50:17.828 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 8.7 MB/s | 19 MB 00:02 2024-03-28T14:50:21.662 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-28T14:50:21.758 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-28T14:50:23.201 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-28T14:50:23.363 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-28T14:50:24.742 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-28T14:50:25.078 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-28T14:50:29.914 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-03-28T14:50:29.992 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-28T14:50:32.088 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T14:50:32.088 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T14:50:32.098 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T14:50:32.098 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T14:50:32.175 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-28T14:50:32.183 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-28T14:50:32.183 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-03-28T14:50:32.183 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-28T14:50:32.183 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-28T14:50:32.183 INFO:teuthology.orchestra.run.smithi186.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-28T14:50:32.183 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-28T14:50:32.183 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-28T14:50:32.183 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-28T14:50:32.183 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-28T14:50:32.183 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:50:32.183 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-28T14:50:32.184 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-28T14:50:32.184 INFO:teuthology.orchestra.run.smithi186.stdout:Install 4 Packages 2024-03-28T14:50:32.184 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:50:32.184 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 76 M 2024-03-28T14:50:32.184 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 123 M 2024-03-28T14:50:32.184 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-28T14:50:32.192 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-28T14:50:32.196 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-28T14:50:32.197 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-03-28T14:50:32.197 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-28T14:50:32.197 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-28T14:50:32.197 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-28T14:50:32.197 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-28T14:50:32.197 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-28T14:50:32.197 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-28T14:50:32.197 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-28T14:50:32.197 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:50:32.197 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-28T14:50:32.198 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-28T14:50:32.198 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-03-28T14:50:32.198 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:50:32.198 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 76 M 2024-03-28T14:50:32.199 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 123 M 2024-03-28T14:50:32.199 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-28T14:50:32.376 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 576 kB/s | 66 kB 00:00 2024-03-28T14:50:32.415 INFO:teuthology.orchestra.run.smithi186.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 667 kB/s | 66 kB 00:00 2024-03-28T14:50:32.877 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-03-28T14:50:33.235 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 32 MB/s | 27 MB 00:00 2024-03-28T14:50:33.458 INFO:teuthology.orchestra.run.smithi186.stdout:(2/4): kernel-modules-core-5.14.0-432.el9.x86_6 26 MB/s | 27 MB 00:01 2024-03-28T14:50:33.527 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 26 MB/s | 33 MB 00:01 2024-03-28T14:50:33.528 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-28T14:50:33.528 INFO:teuthology.orchestra.run.smithi191.stdout:Total 57 MB/s | 76 MB 00:01 2024-03-28T14:50:33.941 INFO:teuthology.orchestra.run.smithi186.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 21 MB/s | 33 MB 00:01 2024-03-28T14:50:34.196 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-28T14:50:34.458 INFO:teuthology.orchestra.run.smithi186.stdout:(4/4): kernel-core-5.14.0-432.el9.x86_64.rpm 7.0 MB/s | 15 MB 00:02 2024-03-28T14:50:34.458 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-28T14:50:34.458 INFO:teuthology.orchestra.run.smithi186.stdout:Total 33 MB/s | 76 MB 00:02 2024-03-28T14:50:34.549 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-28T14:50:34.549 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-28T14:50:35.127 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-28T14:50:35.522 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-28T14:50:35.522 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-28T14:50:35.564 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-28T14:50:35.564 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-28T14:50:36.519 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-28T14:50:36.520 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-28T14:50:38.022 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-28T14:50:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-28T14:50:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T14:50:39.001 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-28T14:50:39.618 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-28T14:50:39.707 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T14:50:41.472 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T14:50:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T14:50:42.466 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T14:50:42.602 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T14:50:46.342 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T14:50:46.441 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:50:47.315 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T14:50:47.351 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:50:51.176 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:50:52.107 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:51:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:51:20.589 INFO:teuthology.orchestra.run.smithi191.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T14:51:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:51:20.633 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:51:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:51:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-28T14:51:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T14:51:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T14:51:21.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:51:21.485 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:51:21.485 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-28T14:51:21.485 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-28T14:51:21.485 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-28T14:51:21.485 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-28T14:51:21.485 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-28T14:51:21.485 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:51:21.485 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-28T14:51:21.502 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:51:21.502 INFO:teuthology.orchestra.run.smithi186.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T14:51:21.502 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:51:21.547 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:51:21.812 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T14:51:21.875 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:51:21.875 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-28T14:51:21.875 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T14:51:21.875 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T14:51:21.907 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T14:51:21.908 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T14:51:21.908 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-28T14:51:21.908 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T14:51:21.908 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T14:51:21.908 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-28T14:51:21.908 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T14:51:21.908 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T14:51:21.989 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T14:51:21.989 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T14:51:21.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2024-03-28T14:51:22.023 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2024-03-28T14:51:22.023 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2024-03-28T14:51:22.023 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.06 2024-03-28T14:51:22.023 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 68.el9 2024-03-28T14:51:22.023 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2024-03-28T14:51:22.023 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T14:51:22.023 INFO:teuthology.orchestra.run.smithi191.stdout:Group : Unspecified 2024-03-28T14:51:22.023 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 8264254 2024-03-28T14:51:22.023 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : builder@centos.org 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2024-03-28T14:51:22.024 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:51:22.025 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T14:51:22.025 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T14:51:22.025 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 5.14.0-432.el9.x86_64 2024-03-28T14:51:22.026 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T14:51:22.293 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T14:51:22.293 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:51:22.293 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-28T14:51:22.293 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-28T14:51:22.293 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-28T14:51:22.294 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-28T14:51:22.294 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-28T14:51:22.294 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:51:22.294 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-28T14:51:22.630 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T14:51:22.668 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2024-03-28T14:51:22.771 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T14:51:22.771 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T14:51:22.771 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-28T14:51:22.772 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T14:51:22.772 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T14:51:22.772 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-28T14:51:22.772 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T14:51:22.772 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T14:51:22.851 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T14:51:22.851 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T14:51:22.852 DEBUG:teuthology.orchestra.run.smithi186:> sudo rpm -qi grub2-tools 2024-03-28T14:51:22.895 INFO:teuthology.orchestra.run.smithi186.stdout:Name : grub2-tools 2024-03-28T14:51:22.895 INFO:teuthology.orchestra.run.smithi186.stdout:Epoch : 1 2024-03-28T14:51:22.895 INFO:teuthology.orchestra.run.smithi186.stdout:Version : 2.06 2024-03-28T14:51:22.895 INFO:teuthology.orchestra.run.smithi186.stdout:Release : 68.el9 2024-03-28T14:51:22.895 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: x86_64 2024-03-28T14:51:22.895 INFO:teuthology.orchestra.run.smithi186.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T14:51:22.895 INFO:teuthology.orchestra.run.smithi186.stdout:Group : Unspecified 2024-03-28T14:51:22.895 INFO:teuthology.orchestra.run.smithi186.stdout:Size : 8264254 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:License : GPLv3+ 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:Packager : builder@centos.org 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:Vendor : CentOS 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:Summary : Support tools for GRUB. 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:Description : 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T14:51:22.896 INFO:teuthology.orchestra.run.smithi186.stdout:hardware devices. 2024-03-28T14:51:22.897 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:51:22.897 INFO:teuthology.orchestra.run.smithi186.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T14:51:22.898 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T14:51:22.898 INFO:teuthology.task.kernel:Updating grub on smithi186 to boot 5.14.0-432.el9.x86_64 2024-03-28T14:51:22.898 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T14:51:23.513 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2024-03-28T14:51:24.014 INFO:teuthology.orchestra.run.smithi191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T14:51:24.044 INFO:teuthology.orchestra.run.smithi191.stderr:done 2024-03-28T14:51:24.047 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-03-28T14:51:24.061 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.HbmoSPWnw2 2024-03-28T14:51:24.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HbmoSPWnw2 2024-03-28T14:51:24.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.HbmoSPWnw2 2024-03-28T14:51:24.271 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.HbmoSPWnw2 is 6KB 2024-03-28T14:51:24.321 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.HbmoSPWnw2 2024-03-28T14:51:24.335 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T14:51:24.400 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T14:51:24.400 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T14:51:24.400 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T14:51:24.400 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-28T14:51:24.402 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-28T14:51:24.540 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2024-03-28T14:51:24.846 INFO:teuthology.orchestra.run.smithi186.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T14:51:24.884 INFO:teuthology.orchestra.run.smithi186.stderr:done 2024-03-28T14:51:24.886 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-03-28T14:51:24.901 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.ZZjWDC8Gea 2024-03-28T14:51:24.902 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZZjWDC8Gea 2024-03-28T14:51:24.977 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.ZZjWDC8Gea 2024-03-28T14:51:25.126 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.ZZjWDC8Gea is 6KB 2024-03-28T14:51:25.184 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.ZZjWDC8Gea 2024-03-28T14:51:25.198 DEBUG:teuthology.orchestra.run.smithi186:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T14:51:25.270 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T14:51:25.270 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T14:51:25.270 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T14:51:25.270 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-28T14:51:25.271 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-28T14:51:25.432 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2024-03-28T14:51:54.571 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-28T14:51:54.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-28T14:51:54.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:51:55.436 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-28T14:51:55.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-28T14:51:55.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:52:12.962 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-03-28T14:52:13.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-03-28T14:52:21.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-28T14:52:21.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:52:22.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-28T14:52:22.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:52:25.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-03-28T14:52:26.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-03-28T14:52:37.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-28T14:52:37.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:52:38.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-28T14:52:38.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:53:08.908 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-28T14:53:09.336 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-28T14:53:09.336 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-28T14:53:09.336 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-03-28T14:53:09.350 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-432.el9.x86_64 2024-03-28T14:53:09.350 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-28T14:53:09.350 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T14:53:09.351 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T14:53:10.351 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-28T14:53:10.351 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T14:53:10.439 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2024-03-28T14:53:10.477 DEBUG:teuthology.parallel:result is None 2024-03-28T14:53:38.052 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T14:53:53.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-28T14:53:53.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:53:53.387 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-03-28T14:53:53.783 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-28T14:53:53.783 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-28T14:53:53.783 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-03-28T14:53:53.798 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-432.el9.x86_64 2024-03-28T14:53:53.799 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-28T14:53:53.799 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T14:53:53.799 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T14:53:54.799 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-28T14:53:54.799 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T14:53:54.863 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2024-03-28T14:53:54.907 DEBUG:teuthology.parallel:result is None 2024-03-28T14:53:54.907 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T14:53:54.915 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T14:53:54.915 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T14:53:54.917 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T14:53:54.936 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T14:53:54.951 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T14:53:54.983 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T14:53:54.984 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T14:53:54.986 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T14:53:55.015 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T14:53:55.022 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T14:53:55.022 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T14:53:55.052 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T14:53:55.083 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T14:53:55.090 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T14:53:55.096 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T14:53:55.104 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T14:53:55.106 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T14:53:55.113 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T14:53:55.114 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T14:53:55.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T14:53:55.177 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T14:53:55.188 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T14:53:55.188 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T14:53:55.210 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T14:53:55.233 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T14:53:55.305 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T14:53:55.350 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T14:53:55.436 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T14:53:55.475 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T14:53:55.475 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T14:53:55.543 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T14:53:55.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T14:53:55.642 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T14:53:55.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T14:53:55.769 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T14:53:55.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T14:53:55.837 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-03-28T14:53:55.840 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-03-28T14:53:55.893 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T14:53:55.941 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T14:53:56.358 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T14:53:56.366 INFO:teuthology.task.internal:Starting timer... 2024-03-28T14:53:56.366 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T14:53:56.416 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T14:53:56.445 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-28T14:53:56.446 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-03-28T14:53:56.493 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-03-28T14:53:56.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-03-28T14:53:56.562 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-03-28T14:53:56.564 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T14:53:56.564 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-03-28T14:53:56.586 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-03-28T14:53:56.586 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-03-28T14:53:56.621 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-03-28T14:53:56.621 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi186.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-03-28T14:53:56.622 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T14:53:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:53:56.655 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T14:53:56.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:53:56.690 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T14:53:56.690 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-03-28T14:53:56.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-03-28T14:53:56.763 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T14:53:56.773 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-28T14:53:56.779 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-28T14:53:56.902 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T14:53:56.916 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T14:53:56.917 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi186.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T14:58:31.556 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi186.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-03-28T14:58:31.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-28T14:58:31.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:58:31.653 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-03-28T14:58:31.740 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-28T14:58:31.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-28T14:58:31.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:58:31.877 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-28T14:58:31.955 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-28T14:58:31.956 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T14:58:31.968 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T14:58:31.968 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T14:58:31.968 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T14:58:31.974 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T14:58:31.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T14:58:32.002 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T14:58:32.018 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T14:58:32.029 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T14:58:32.047 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T14:58:32.048 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-03-28T14:58:32.061 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-03-28T14:58:32.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-03-28T14:58:32.077 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T14:58:32.089 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-03-28T14:58:32.094 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T14:58:32.106 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T14:58:32.122 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T14:58:32.143 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-03-28T14:58:32.147 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T14:58:32.148 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-03-28T14:58:32.148 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T14:58:32.148 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T14:58:32.148 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T14:58:32.148 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T14:58:32.174 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-28T14:58:32.177 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T14:58:32.178 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-28T14:58:32.178 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T14:58:32.178 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T14:58:32.178 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T14:58:32.178 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T14:58:32.179 INFO:teuthology.run_tasks:Running task install... 2024-03-28T14:58:32.189 DEBUG:teuthology.task.install:project ceph 2024-03-28T14:58:32.189 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'}} 2024-03-28T14:58:32.189 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'} 2024-03-28T14:58:32.190 INFO:teuthology.task.install:Using flavor: default 2024-03-28T14:58:32.195 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T14:58:32.195 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T14:58:32.196 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15', 'tag': None, 'wait_for_package': False} 2024-03-28T14:58:32.196 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T14:58:32.198 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15', 'tag': None, 'wait_for_package': False} 2024-03-28T14:58:32.198 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T14:58:32.341 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/centos/9/flavors/default/ 2024-03-28T14:58:32.341 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2499.ge0e48ba6 2024-03-28T14:58:32.414 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/centos/9/flavors/default/ 2024-03-28T14:58:32.414 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2499.ge0e48ba6 2024-03-28T14:58:32.477 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T14:58:32.477 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T14:58:32.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T14:58:32.506 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T14:58:32.506 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T14:58:32.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T14:58:32.601 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T14:58:32.601 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T14:58:32.601 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T14:58:32.629 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T14:58:32.629 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T14:58:32.656 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T14:58:32.698 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T14:58:32.723 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-03-28T14:58:32.725 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-03-28T14:58:32.778 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T14:58:32.845 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-03-28T14:58:32.847 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-03-28T14:58:32.972 INFO:teuthology.orchestra.run.smithi191.stdout:74 files removed 2024-03-28T14:58:33.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T14:58:33.087 INFO:teuthology.orchestra.run.smithi186.stdout:74 files removed 2024-03-28T14:58:33.113 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T14:58:33.718 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 272 kB/s | 83 kB 00:00 2024-03-28T14:58:33.934 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 132 kB/s | 83 kB 00:00 2024-03-28T14:58:34.168 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-03-28T14:58:34.377 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-03-28T14:58:34.485 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-03-28T14:58:34.886 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-03-28T14:58:34.988 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-28T14:58:35.356 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-28T14:58:36.405 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-28T14:58:36.749 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-28T14:58:37.412 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-28T14:58:37.781 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-28T14:58:40.041 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-28T14:58:43.472 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 3.8 MB/s | 19 MB 00:05 2024-03-28T14:58:47.614 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.1 MB 00:00 2024-03-28T14:58:49.382 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-28T14:58:50.442 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-28T14:58:51.458 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 789 kB/s | 6.1 MB 00:07 2024-03-28T14:58:53.276 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-28T14:58:54.170 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-28T14:58:55.358 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-28T14:58:57.853 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T14:58:57.853 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T14:58:57.867 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T14:58:57.868 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T14:58:57.869 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T14:58:57.870 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T14:58:57.872 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T14:58:57.875 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T14:58:57.876 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T14:58:57.957 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-28T14:58:57.990 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-03-28T14:58:57.990 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-28T14:58:57.990 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-03-28T14:58:57.990 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 6.6 k 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 5.4 M 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 890 k 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 145 k 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 1.5 M 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 141 k 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 3.5 M 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 7.4 M 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 49 k 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 9.8 M 2024-03-28T14:58:57.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 48 M 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 264 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 762 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 221 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 27 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 18 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 32 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 746 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 127 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T14:58:57.992 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T14:58:57.993 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T14:58:57.993 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T14:58:57.993 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T14:58:57.993 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T14:58:57.993 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 162 k 2024-03-28T14:58:57.993 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 320 k 2024-03-28T14:58:57.993 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 298 k 2024-03-28T14:58:57.993 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 99 k 2024-03-28T14:58:57.993 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 85 k 2024-03-28T14:58:57.993 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 3.1 M 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 170 k 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T14:58:57.994 INFO:teuthology.orchestra.run.smithi186.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 3.4 M 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 3.1 M 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T14:58:57.995 INFO:teuthology.orchestra.run.smithi186.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T14:58:57.996 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T14:58:57.996 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-28T14:58:57.996 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T14:58:57.996 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T14:58:57.996 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T14:58:57.996 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 21 M 2024-03-28T14:58:57.996 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 34 k 2024-03-28T14:58:57.996 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 2.4 M 2024-03-28T14:58:57.996 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 249 k 2024-03-28T14:58:57.996 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 4.7 M 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 17 M 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 16 k 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 25 k 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 163 k 2024-03-28T14:58:57.997 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 501 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 5.1 M 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T14:58:57.998 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T14:58:57.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 45 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 134 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T14:58:58.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T14:58:58.001 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T14:58:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T14:58:58.003 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-28T14:58:58.004 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-03-28T14:58:58.005 INFO:teuthology.orchestra.run.smithi186.stdout:Install 150 Packages 2024-03-28T14:58:58.005 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 24 Packages 2024-03-28T14:58:58.005 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:58:58.005 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 227 M 2024-03-28T14:58:58.005 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-28T14:58:58.859 INFO:teuthology.orchestra.run.smithi186.stdout:(1/174): ceph-19.0.0-2499.ge0e48ba6.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-03-28T14:58:59.104 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-28T14:58:59.318 INFO:teuthology.orchestra.run.smithi186.stdout:(2/174): ceph-base-19.0.0-2499.ge0e48ba6.el9.x8 7.5 MB/s | 5.4 MB 00:00 2024-03-28T14:58:59.368 INFO:teuthology.orchestra.run.smithi186.stdout:(3/174): ceph-fuse-19.0.0-2499.ge0e48ba6.el9.x8 1.7 MB/s | 890 kB 00:00 2024-03-28T14:58:59.393 INFO:teuthology.orchestra.run.smithi186.stdout:(4/174): ceph-immutable-object-cache-19.0.0-249 1.9 MB/s | 145 kB 00:00 2024-03-28T14:58:59.552 INFO:teuthology.orchestra.run.smithi186.stdout:(5/174): ceph-mgr-19.0.0-2499.ge0e48ba6.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-03-28T14:59:00.010 INFO:teuthology.orchestra.run.smithi186.stdout:(6/174): ceph-mon-19.0.0-2499.ge0e48ba6.el9.x86 10 MB/s | 4.7 MB 00:00 2024-03-28T14:59:00.286 INFO:teuthology.orchestra.run.smithi186.stdout:(7/174): ceph-mds-19.0.0-2499.ge0e48ba6.el9.x86 2.6 MB/s | 2.4 MB 00:00 2024-03-28T14:59:01.553 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T14:59:01.554 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T14:59:01.568 INFO:teuthology.orchestra.run.smithi191.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T14:59:01.569 INFO:teuthology.orchestra.run.smithi191.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T14:59:01.570 INFO:teuthology.orchestra.run.smithi191.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T14:59:01.572 INFO:teuthology.orchestra.run.smithi191.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T14:59:01.573 INFO:teuthology.orchestra.run.smithi191.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T14:59:01.577 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T14:59:01.578 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T14:59:01.657 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-28T14:59:01.675 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-28T14:59:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-28T14:59:01.675 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-28T14:59:01.675 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-28T14:59:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 6.6 k 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 5.4 M 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 890 k 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 145 k 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 1.5 M 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 141 k 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 3.5 M 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 7.4 M 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 49 k 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 9.8 M 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 48 M 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 264 k 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 762 k 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 221 k 2024-03-28T14:59:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-shell x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 27 k 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 18 k 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 32 k 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 746 k 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 127 k 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T14:59:01.677 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 162 k 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 320 k 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 298 k 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 99 k 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 85 k 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 3.1 M 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 170 k 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T14:59:01.678 INFO:teuthology.orchestra.run.smithi191.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T14:59:01.679 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T14:59:01.679 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T14:59:01.679 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T14:59:01.683 INFO:teuthology.orchestra.run.smithi191.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T14:59:01.683 INFO:teuthology.orchestra.run.smithi191.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T14:59:01.683 INFO:teuthology.orchestra.run.smithi191.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T14:59:01.683 INFO:teuthology.orchestra.run.smithi191.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T14:59:01.683 INFO:teuthology.orchestra.run.smithi191.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T14:59:01.683 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 3.4 M 2024-03-28T14:59:01.683 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 3.1 M 2024-03-28T14:59:01.683 INFO:teuthology.orchestra.run.smithi191.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T14:59:01.683 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T14:59:01.683 INFO:teuthology.orchestra.run.smithi191.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 21 M 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 34 k 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 2.4 M 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 249 k 2024-03-28T14:59:01.684 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 4.7 M 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 17 M 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2499.ge0e48ba6.el9 ceph-noarch 16 k 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 25 k 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 163 k 2024-03-28T14:59:01.685 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T14:59:01.686 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T14:59:01.686 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T14:59:01.686 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T14:59:01.686 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T14:59:01.686 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 501 k 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 5.1 M 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T14:59:01.687 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 45 k 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:19.0.0-2499.ge0e48ba6.el9 ceph 134 k 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T14:59:01.688 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T14:59:01.689 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T14:59:01.690 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T14:59:01.691 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T14:59:01.692 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout:Install 150 Packages 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 24 Packages 2024-03-28T14:59:01.693 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:59:01.694 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 227 M 2024-03-28T14:59:01.694 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-28T14:59:01.969 INFO:teuthology.orchestra.run.smithi186.stdout:(8/174): ceph-common-19.0.0-2499.ge0e48ba6.el9. 6.1 MB/s | 21 MB 00:03 2024-03-28T14:59:02.161 INFO:teuthology.orchestra.run.smithi186.stdout:(9/174): ceph-osd-19.0.0-2499.ge0e48ba6.el9.x86 7.8 MB/s | 17 MB 00:02 2024-03-28T14:59:02.186 INFO:teuthology.orchestra.run.smithi186.stdout:(10/174): ceph-selinux-19.0.0-2499.ge0e48ba6.el 116 kB/s | 25 kB 00:00 2024-03-28T14:59:02.295 INFO:teuthology.orchestra.run.smithi186.stdout:(11/174): ceph-radosgw-19.0.0-2499.ge0e48ba6.el 4.9 MB/s | 9.8 MB 00:02 2024-03-28T14:59:02.355 INFO:teuthology.orchestra.run.smithi191.stdout:(1/174): ceph-19.0.0-2499.ge0e48ba6.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-03-28T14:59:02.412 INFO:teuthology.orchestra.run.smithi186.stdout:(12/174): cephfs-shell-19.0.0-2499.ge0e48ba6.el 232 kB/s | 27 kB 00:00 2024-03-28T14:59:02.479 INFO:teuthology.orchestra.run.smithi186.stdout:(13/174): libcephfs-devel-19.0.0-2499.ge0e48ba6 478 kB/s | 32 kB 00:00 2024-03-28T14:59:02.696 INFO:teuthology.orchestra.run.smithi186.stdout:(14/174): cephfs-mirror-19.0.0-2499.ge0e48ba6.e 435 kB/s | 221 kB 00:00 2024-03-28T14:59:02.804 INFO:teuthology.orchestra.run.smithi186.stdout:(15/174): libcephsqlite-19.0.0-2499.ge0e48ba6.e 1.5 MB/s | 163 kB 00:00 2024-03-28T14:59:02.913 INFO:teuthology.orchestra.run.smithi186.stdout:(16/174): librados-devel-19.0.0-2499.ge0e48ba6. 1.1 MB/s | 127 kB 00:00 2024-03-28T14:59:03.046 INFO:teuthology.orchestra.run.smithi186.stdout:(17/174): libcephfs2-19.0.0-2499.ge0e48ba6.el9. 1.3 MB/s | 746 kB 00:00 2024-03-28T14:59:03.197 INFO:teuthology.orchestra.run.smithi186.stdout:(18/174): libradosstriper1-19.0.0-2499.ge0e48ba 1.7 MB/s | 501 kB 00:00 2024-03-28T14:59:03.323 INFO:teuthology.orchestra.run.smithi186.stdout:(19/174): python3-ceph-argparse-19.0.0-2499.ge0 361 kB/s | 45 kB 00:00 2024-03-28T14:59:03.455 INFO:teuthology.orchestra.run.smithi191.stdout:(2/174): ceph-base-19.0.0-2499.ge0e48ba6.el9.x8 3.6 MB/s | 5.4 MB 00:01 2024-03-28T14:59:03.487 INFO:teuthology.orchestra.run.smithi186.stdout:(20/174): python3-ceph-common-19.0.0-2499.ge0e4 818 kB/s | 134 kB 00:00 2024-03-28T14:59:03.530 INFO:teuthology.orchestra.run.smithi191.stdout:(3/174): ceph-immutable-object-cache-19.0.0-249 1.9 MB/s | 145 kB 00:00 2024-03-28T14:59:03.598 INFO:teuthology.orchestra.run.smithi186.stdout:(21/174): python3-cephfs-19.0.0-2499.ge0e48ba6. 1.4 MB/s | 162 kB 00:00 2024-03-28T14:59:03.665 INFO:teuthology.orchestra.run.smithi191.stdout:(4/174): ceph-fuse-19.0.0-2499.ge0e48ba6.el9.x8 680 kB/s | 890 kB 00:01 2024-03-28T14:59:03.890 INFO:teuthology.orchestra.run.smithi186.stdout:(22/174): python3-rados-19.0.0-2499.ge0e48ba6.e 1.1 MB/s | 320 kB 00:00 2024-03-28T14:59:04.098 INFO:teuthology.orchestra.run.smithi186.stdout:(23/174): python3-rbd-19.0.0-2499.ge0e48ba6.el9 1.4 MB/s | 298 kB 00:00 2024-03-28T14:59:04.223 INFO:teuthology.orchestra.run.smithi186.stdout:(24/174): python3-rgw-19.0.0-2499.ge0e48ba6.el9 796 kB/s | 99 kB 00:00 2024-03-28T14:59:04.257 INFO:teuthology.orchestra.run.smithi191.stdout:(5/174): ceph-mds-19.0.0-2499.ge0e48ba6.el9.x86 3.2 MB/s | 2.4 MB 00:00 2024-03-28T14:59:04.324 INFO:teuthology.orchestra.run.smithi186.stdout:(25/174): rbd-fuse-19.0.0-2499.ge0e48ba6.el9.x8 854 kB/s | 85 kB 00:00 2024-03-28T14:59:05.032 INFO:teuthology.orchestra.run.smithi186.stdout:(26/174): librgw2-19.0.0-2499.ge0e48ba6.el9.x86 2.6 MB/s | 5.1 MB 00:01 2024-03-28T14:59:05.332 INFO:teuthology.orchestra.run.smithi191.stdout:(6/174): ceph-mgr-19.0.0-2499.ge0e48ba6.el9.x86 918 kB/s | 1.5 MB 00:01 2024-03-28T14:59:05.491 INFO:teuthology.orchestra.run.smithi186.stdout:(27/174): rbd-nbd-19.0.0-2499.ge0e48ba6.el9.x86 372 kB/s | 170 kB 00:00 2024-03-28T14:59:05.599 INFO:teuthology.orchestra.run.smithi186.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2499.g 314 kB/s | 34 kB 00:00 2024-03-28T14:59:05.666 INFO:teuthology.orchestra.run.smithi191.stdout:(7/174): ceph-common-19.0.0-2499.ge0e48ba6.el9. 5.6 MB/s | 21 MB 00:03 2024-03-28T14:59:05.725 INFO:teuthology.orchestra.run.smithi186.stdout:(29/174): rbd-mirror-19.0.0-2499.ge0e48ba6.el9. 2.2 MB/s | 3.1 MB 00:01 2024-03-28T14:59:05.741 INFO:teuthology.orchestra.run.smithi191.stdout:(8/174): ceph-mon-19.0.0-2499.ge0e48ba6.el9.x86 3.2 MB/s | 4.7 MB 00:01 2024-03-28T14:59:05.750 INFO:teuthology.orchestra.run.smithi186.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2499.ge0e48ba 942 kB/s | 141 kB 00:00 2024-03-28T14:59:05.825 INFO:teuthology.orchestra.run.smithi191.stdout:(9/174): ceph-selinux-19.0.0-2499.ge0e48ba6.el9 301 kB/s | 25 kB 00:00 2024-03-28T14:59:07.167 INFO:teuthology.orchestra.run.smithi191.stdout:(10/174): ceph-radosgw-19.0.0-2499.ge0e48ba6.el 6.6 MB/s | 9.8 MB 00:01 2024-03-28T14:59:07.217 INFO:teuthology.orchestra.run.smithi191.stdout:(11/174): cephfs-mirror-19.0.0-2499.ge0e48ba6.e 4.3 MB/s | 221 kB 00:00 2024-03-28T14:59:07.267 INFO:teuthology.orchestra.run.smithi191.stdout:(12/174): cephfs-shell-19.0.0-2499.ge0e48ba6.el 540 kB/s | 27 kB 00:00 2024-03-28T14:59:07.309 INFO:teuthology.orchestra.run.smithi191.stdout:(13/174): libcephfs-devel-19.0.0-2499.ge0e48ba6 759 kB/s | 32 kB 00:00 2024-03-28T14:59:07.410 INFO:teuthology.orchestra.run.smithi191.stdout:(14/174): libcephfs2-19.0.0-2499.ge0e48ba6.el9. 7.3 MB/s | 746 kB 00:00 2024-03-28T14:59:07.460 INFO:teuthology.orchestra.run.smithi191.stdout:(15/174): libcephsqlite-19.0.0-2499.ge0e48ba6.e 3.2 MB/s | 163 kB 00:00 2024-03-28T14:59:07.535 INFO:teuthology.orchestra.run.smithi191.stdout:(16/174): librados-devel-19.0.0-2499.ge0e48ba6. 1.7 MB/s | 127 kB 00:00 2024-03-28T14:59:07.628 INFO:teuthology.orchestra.run.smithi191.stdout:(17/174): libradosstriper1-19.0.0-2499.ge0e48ba 5.3 MB/s | 501 kB 00:00 2024-03-28T14:59:08.245 INFO:teuthology.orchestra.run.smithi191.stdout:(18/174): librgw2-19.0.0-2499.ge0e48ba6.el9.x86 8.3 MB/s | 5.1 MB 00:00 2024-03-28T14:59:08.303 INFO:teuthology.orchestra.run.smithi191.stdout:(19/174): python3-ceph-argparse-19.0.0-2499.ge0 776 kB/s | 45 kB 00:00 2024-03-28T14:59:08.370 INFO:teuthology.orchestra.run.smithi191.stdout:(20/174): python3-ceph-common-19.0.0-2499.ge0e4 2.0 MB/s | 134 kB 00:00 2024-03-28T14:59:08.437 INFO:teuthology.orchestra.run.smithi191.stdout:(21/174): python3-cephfs-19.0.0-2499.ge0e48ba6. 2.4 MB/s | 162 kB 00:00 2024-03-28T14:59:08.513 INFO:teuthology.orchestra.run.smithi191.stdout:(22/174): python3-rados-19.0.0-2499.ge0e48ba6.e 4.2 MB/s | 320 kB 00:00 2024-03-28T14:59:08.588 INFO:teuthology.orchestra.run.smithi191.stdout:(23/174): python3-rbd-19.0.0-2499.ge0e48ba6.el9 3.9 MB/s | 298 kB 00:00 2024-03-28T14:59:08.655 INFO:teuthology.orchestra.run.smithi191.stdout:(24/174): python3-rgw-19.0.0-2499.ge0e48ba6.el9 1.5 MB/s | 99 kB 00:00 2024-03-28T14:59:08.722 INFO:teuthology.orchestra.run.smithi191.stdout:(25/174): rbd-fuse-19.0.0-2499.ge0e48ba6.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-28T14:59:09.115 INFO:teuthology.orchestra.run.smithi191.stdout:(26/174): rbd-mirror-19.0.0-2499.ge0e48ba6.el9. 7.9 MB/s | 3.1 MB 00:00 2024-03-28T14:59:09.174 INFO:teuthology.orchestra.run.smithi191.stdout:(27/174): rbd-nbd-19.0.0-2499.ge0e48ba6.el9.x86 2.9 MB/s | 170 kB 00:00 2024-03-28T14:59:09.232 INFO:teuthology.orchestra.run.smithi191.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2499.g 582 kB/s | 34 kB 00:00 2024-03-28T14:59:09.250 INFO:teuthology.orchestra.run.smithi186.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2499.ge0e48 1.0 MB/s | 3.5 MB 00:03 2024-03-28T14:59:09.291 INFO:teuthology.orchestra.run.smithi191.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2499.ge0e48ba 2.4 MB/s | 141 kB 00:00 2024-03-28T14:59:09.617 INFO:teuthology.orchestra.run.smithi186.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2499.ge0 678 kB/s | 249 kB 00:00 2024-03-28T14:59:09.742 INFO:teuthology.orchestra.run.smithi186.stdout:(33/174): ceph-mgr-rook-19.0.0-2499.ge0e48ba6.e 395 kB/s | 49 kB 00:00 2024-03-28T14:59:09.758 INFO:teuthology.orchestra.run.smithi191.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2499.ge0e48 7.6 MB/s | 3.5 MB 00:00 2024-03-28T14:59:09.809 INFO:teuthology.orchestra.run.smithi186.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2499.ge 247 kB/s | 16 kB 00:00 2024-03-28T14:59:10.234 INFO:teuthology.orchestra.run.smithi186.stdout:(35/174): ceph-volume-19.0.0-2499.ge0e48ba6.el9 620 kB/s | 264 kB 00:00 2024-03-28T14:59:10.867 INFO:teuthology.orchestra.run.smithi191.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 6.7 MB/s | 7.4 MB 00:01 2024-03-28T14:59:10.951 INFO:teuthology.orchestra.run.smithi191.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2499.ge0 2.9 MB/s | 249 kB 00:00 2024-03-28T14:59:11.063 INFO:teuthology.orchestra.run.smithi191.stdout:(33/174): ceph-mgr-rook-19.0.0-2499.ge0e48ba6.e 441 kB/s | 49 kB 00:00 2024-03-28T14:59:11.126 INFO:teuthology.orchestra.run.smithi186.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 1.4 MB/s | 7.4 MB 00:05 2024-03-28T14:59:11.143 INFO:teuthology.orchestra.run.smithi191.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2499.ge 206 kB/s | 16 kB 00:00 2024-03-28T14:59:11.176 INFO:teuthology.orchestra.run.smithi186.stdout:(37/174): cephadm-19.0.0-2499.ge0e48ba6.el9.noa 809 kB/s | 762 kB 00:00 2024-03-28T14:59:11.227 INFO:teuthology.orchestra.run.smithi191.stdout:(35/174): ceph-volume-19.0.0-2499.ge0e48ba6.el9 3.1 MB/s | 264 kB 00:00 2024-03-28T14:59:11.227 INFO:teuthology.orchestra.run.smithi186.stdout:(38/174): cephfs-top-19.0.0-2499.ge0e48ba6.el9. 182 kB/s | 18 kB 00:00 2024-03-28T14:59:11.327 INFO:teuthology.orchestra.run.smithi186.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-03-28T14:59:11.361 INFO:teuthology.orchestra.run.smithi186.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 6.8 MB/s | 920 kB 00:00 2024-03-28T14:59:11.385 INFO:teuthology.orchestra.run.smithi191.stdout:(36/174): cephadm-19.0.0-2499.ge0e48ba6.el9.noa 4.7 MB/s | 762 kB 00:00 2024-03-28T14:59:11.386 INFO:teuthology.orchestra.run.smithi186.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-28T14:59:11.411 INFO:teuthology.orchestra.run.smithi186.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-28T14:59:11.428 INFO:teuthology.orchestra.run.smithi186.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 766 kB/s | 32 kB 00:00 2024-03-28T14:59:11.453 INFO:teuthology.orchestra.run.smithi191.stdout:(37/174): cephfs-top-19.0.0-2499.ge0e48ba6.el9. 273 kB/s | 18 kB 00:00 2024-03-28T14:59:11.504 INFO:teuthology.orchestra.run.smithi186.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-28T14:59:11.549 INFO:teuthology.orchestra.run.smithi186.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 145 kB/s | 17 kB 00:00 2024-03-28T14:59:11.571 INFO:teuthology.orchestra.run.smithi186.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 151 kB/s | 10 kB 00:00 2024-03-28T14:59:11.595 INFO:teuthology.orchestra.run.smithi191.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-28T14:59:11.596 INFO:teuthology.orchestra.run.smithi186.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.3 MB/s | 159 kB 00:00 2024-03-28T14:59:11.621 INFO:teuthology.orchestra.run.smithi186.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-28T14:59:11.647 INFO:teuthology.orchestra.run.smithi186.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-28T14:59:11.662 INFO:teuthology.orchestra.run.smithi191.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 13 MB/s | 920 kB 00:00 2024-03-28T14:59:11.672 INFO:teuthology.orchestra.run.smithi186.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-28T14:59:11.697 INFO:teuthology.orchestra.run.smithi186.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 46 kB 00:00 2024-03-28T14:59:11.704 INFO:teuthology.orchestra.run.smithi191.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-28T14:59:11.723 INFO:teuthology.orchestra.run.smithi186.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T14:59:11.746 INFO:teuthology.orchestra.run.smithi191.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-28T14:59:11.749 INFO:teuthology.orchestra.run.smithi186.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-28T14:59:11.773 INFO:teuthology.orchestra.run.smithi186.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-28T14:59:11.779 INFO:teuthology.orchestra.run.smithi191.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 952 kB/s | 32 kB 00:00 2024-03-28T14:59:11.799 INFO:teuthology.orchestra.run.smithi186.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 689 kB/s | 34 kB 00:00 2024-03-28T14:59:11.924 INFO:teuthology.orchestra.run.smithi186.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 32 MB/s | 4.7 MB 00:00 2024-03-28T14:59:11.930 INFO:teuthology.orchestra.run.smithi191.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-03-28T14:59:11.957 INFO:teuthology.orchestra.run.smithi186.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.5 MB/s | 1.0 MB 00:00 2024-03-28T14:59:11.964 INFO:teuthology.orchestra.run.smithi191.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 515 kB/s | 17 kB 00:00 2024-03-28T14:59:11.982 INFO:teuthology.orchestra.run.smithi186.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T14:59:11.997 INFO:teuthology.orchestra.run.smithi191.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 301 kB/s | 10 kB 00:00 2024-03-28T14:59:12.056 INFO:teuthology.orchestra.run.smithi191.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-28T14:59:12.081 INFO:teuthology.orchestra.run.smithi191.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-28T14:59:12.108 INFO:teuthology.orchestra.run.smithi186.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-03-28T14:59:12.125 INFO:teuthology.orchestra.run.smithi186.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-03-28T14:59:12.132 INFO:teuthology.orchestra.run.smithi191.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-28T14:59:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 536 kB/s | 18 kB 00:00 2024-03-28T14:59:12.167 INFO:teuthology.orchestra.run.smithi186.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-28T14:59:12.183 INFO:teuthology.orchestra.run.smithi191.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-28T14:59:12.184 INFO:teuthology.orchestra.run.smithi186.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 760 kB/s | 32 kB 00:00 2024-03-28T14:59:12.242 INFO:teuthology.orchestra.run.smithi191.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 793 kB/s | 46 kB 00:00 2024-03-28T14:59:12.284 INFO:teuthology.orchestra.run.smithi191.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-28T14:59:12.310 INFO:teuthology.orchestra.run.smithi186.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-28T14:59:12.325 INFO:teuthology.orchestra.run.smithi191.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-28T14:59:12.335 INFO:teuthology.orchestra.run.smithi186.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-28T14:59:12.359 INFO:teuthology.orchestra.run.smithi191.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.3 MB/s | 217 kB 00:00 2024-03-28T14:59:12.360 INFO:teuthology.orchestra.run.smithi186.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-28T14:59:12.386 INFO:teuthology.orchestra.run.smithi186.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-28T14:59:12.393 INFO:teuthology.orchestra.run.smithi191.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-28T14:59:12.402 INFO:teuthology.orchestra.run.smithi186.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 987 kB/s | 41 kB 00:00 2024-03-28T14:59:12.428 INFO:teuthology.orchestra.run.smithi186.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-28T14:59:12.453 INFO:teuthology.orchestra.run.smithi186.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-28T14:59:12.478 INFO:teuthology.orchestra.run.smithi186.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-28T14:59:12.495 INFO:teuthology.orchestra.run.smithi186.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T14:59:12.512 INFO:teuthology.orchestra.run.smithi186.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-28T14:59:12.519 INFO:teuthology.orchestra.run.smithi191.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-28T14:59:12.602 INFO:teuthology.orchestra.run.smithi191.stdout:(56/174): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-03-28T14:59:12.621 INFO:teuthology.orchestra.run.smithi186.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 362 kB/s | 39 kB 00:00 2024-03-28T14:59:12.661 INFO:teuthology.orchestra.run.smithi191.stdout:(57/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T14:59:12.803 INFO:teuthology.orchestra.run.smithi191.stdout:(58/174): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-03-28T14:59:12.829 INFO:teuthology.orchestra.run.smithi186.stdout:(75/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-28T14:59:12.837 INFO:teuthology.orchestra.run.smithi191.stdout:(59/174): python3-jsonpatch-1.21-16.el9.noarch. 733 kB/s | 24 kB 00:00 2024-03-28T14:59:12.855 INFO:teuthology.orchestra.run.smithi186.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 178 kB/s | 41 kB 00:00 2024-03-28T14:59:12.871 INFO:teuthology.orchestra.run.smithi191.stdout:(60/174): python3-jsonpointer-2.0-4.el9.noarch. 541 kB/s | 18 kB 00:00 2024-03-28T14:59:12.880 INFO:teuthology.orchestra.run.smithi186.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-28T14:59:12.904 INFO:teuthology.orchestra.run.smithi191.stdout:(61/174): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-28T14:59:12.905 INFO:teuthology.orchestra.run.smithi186.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-28T14:59:12.930 INFO:teuthology.orchestra.run.smithi186.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T14:59:12.938 INFO:teuthology.orchestra.run.smithi191.stdout:(62/174): python3-markupsafe-1.1.1-12.el9.x86_6 946 kB/s | 32 kB 00:00 2024-03-28T14:59:12.956 INFO:teuthology.orchestra.run.smithi186.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-28T14:59:12.981 INFO:teuthology.orchestra.run.smithi186.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-28T14:59:13.014 INFO:teuthology.orchestra.run.smithi186.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-28T14:59:13.048 INFO:teuthology.orchestra.run.smithi186.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-28T14:59:13.063 INFO:teuthology.orchestra.run.smithi191.stdout:(63/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-28T14:59:13.074 INFO:teuthology.orchestra.run.smithi186.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-28T14:59:13.097 INFO:teuthology.orchestra.run.smithi191.stdout:(64/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 6.4 MB/s | 217 kB 00:00 2024-03-28T14:59:13.099 INFO:teuthology.orchestra.run.smithi186.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-28T14:59:13.124 INFO:teuthology.orchestra.run.smithi186.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-28T14:59:13.131 INFO:teuthology.orchestra.run.smithi191.stdout:(65/174): python3-oauthlib-3.1.1-5.el9.noarch.r 5.1 MB/s | 174 kB 00:00 2024-03-28T14:59:13.165 INFO:teuthology.orchestra.run.smithi191.stdout:(66/174): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-28T14:59:13.198 INFO:teuthology.orchestra.run.smithi191.stdout:(67/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-28T14:59:13.233 INFO:teuthology.orchestra.run.smithi186.stdout:(87/174): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-28T14:59:13.241 INFO:teuthology.orchestra.run.smithi191.stdout:(68/174): python3-protobuf-3.14.0-13.el9.noarch 5.5 MB/s | 238 kB 00:00 2024-03-28T14:59:13.275 INFO:teuthology.orchestra.run.smithi191.stdout:(69/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.9 MB/s | 132 kB 00:00 2024-03-28T14:59:13.308 INFO:teuthology.orchestra.run.smithi191.stdout:(70/174): python3-pyasn1-modules-0.4.8-6.el9.no 6.2 MB/s | 211 kB 00:00 2024-03-28T14:59:13.334 INFO:teuthology.orchestra.run.smithi191.stdout:(71/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-28T14:59:13.350 INFO:teuthology.orchestra.run.smithi186.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-03-28T14:59:13.367 INFO:teuthology.orchestra.run.smithi191.stdout:(72/174): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-28T14:59:13.425 INFO:teuthology.orchestra.run.smithi186.stdout:(89/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-03-28T14:59:13.451 INFO:teuthology.orchestra.run.smithi186.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-03-28T14:59:13.651 INFO:teuthology.orchestra.run.smithi186.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 95 kB/s | 21 kB 00:00 2024-03-28T14:59:13.676 INFO:teuthology.orchestra.run.smithi186.stdout:(92/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 113 kB/s | 25 kB 00:00 2024-03-28T14:59:13.710 INFO:teuthology.orchestra.run.smithi191.stdout:(73/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-28T14:59:13.743 INFO:teuthology.orchestra.run.smithi191.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-28T14:59:13.777 INFO:teuthology.orchestra.run.smithi191.stdout:(75/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-28T14:59:13.827 INFO:teuthology.orchestra.run.smithi191.stdout:(76/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-28T14:59:13.877 INFO:teuthology.orchestra.run.smithi186.stdout:(93/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 18 MB/s | 4.1 MB 00:00 2024-03-28T14:59:13.918 INFO:teuthology.orchestra.run.smithi186.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 6.6 MB/s | 1.6 MB 00:00 2024-03-28T14:59:13.962 INFO:teuthology.orchestra.run.smithi191.stdout:(77/174): uuid-1.6.2-55.el9.x86_64.rpm 418 kB/s | 56 kB 00:00 2024-03-28T14:59:13.985 INFO:teuthology.orchestra.run.smithi186.stdout:(95/174): postgresql-contrib-13.14-1.el9.x86_64 8.2 MB/s | 911 kB 00:00 2024-03-28T14:59:14.011 INFO:teuthology.orchestra.run.smithi186.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-03-28T14:59:14.086 INFO:teuthology.orchestra.run.smithi186.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-03-28T14:59:14.095 INFO:teuthology.orchestra.run.smithi191.stdout:(78/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 474 kB/s | 63 kB 00:00 2024-03-28T14:59:14.137 INFO:teuthology.orchestra.run.smithi191.stdout:(79/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-28T14:59:14.171 INFO:teuthology.orchestra.run.smithi191.stdout:(80/174): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-28T14:59:14.186 INFO:teuthology.orchestra.run.smithi186.stdout:(98/174): postgresql-server-13.14-1.el9.x86_64. 29 MB/s | 5.8 MB 00:00 2024-03-28T14:59:14.211 INFO:teuthology.orchestra.run.smithi186.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-03-28T14:59:14.213 INFO:teuthology.orchestra.run.smithi191.stdout:(81/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-28T14:59:14.262 INFO:teuthology.orchestra.run.smithi186.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-03-28T14:59:14.272 INFO:teuthology.orchestra.run.smithi191.stdout:(82/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-28T14:59:14.305 INFO:teuthology.orchestra.run.smithi191.stdout:(83/174): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-28T14:59:14.337 INFO:teuthology.orchestra.run.smithi186.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 286 kB/s | 36 kB 00:00 2024-03-28T14:59:14.339 INFO:teuthology.orchestra.run.smithi191.stdout:(84/174): python3-pycparser-2.20-6.el9.noarch.r 3.6 MB/s | 124 kB 00:00 2024-03-28T14:59:14.379 INFO:teuthology.orchestra.run.smithi186.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 405 kB/s | 47 kB 00:00 2024-03-28T14:59:14.381 INFO:teuthology.orchestra.run.smithi191.stdout:(85/174): python3-urllib3-1.26.5-5.el9.noarch.r 4.4 MB/s | 187 kB 00:00 2024-03-28T14:59:14.404 INFO:teuthology.orchestra.run.smithi186.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-28T14:59:14.446 INFO:teuthology.orchestra.run.smithi186.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 333 kB/s | 22 kB 00:00 2024-03-28T14:59:14.505 INFO:teuthology.orchestra.run.smithi186.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 8.4 MB/s | 863 kB 00:00 2024-03-28T14:59:14.507 INFO:teuthology.orchestra.run.smithi191.stdout:(86/174): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-28T14:59:14.607 INFO:teuthology.orchestra.run.smithi191.stdout:(87/174): libgfortran-11.4.1-3.el9.x86_64.rpm 7.8 MB/s | 803 kB 00:00 2024-03-28T14:59:14.650 INFO:teuthology.orchestra.run.smithi191.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-03-28T14:59:14.692 INFO:teuthology.orchestra.run.smithi191.stdout:(89/174): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-03-28T14:59:14.784 INFO:teuthology.orchestra.run.smithi191.stdout:(90/174): elfutils-libelf-devel-0.190-2.el9.x86 234 kB/s | 21 kB 00:00 2024-03-28T14:59:14.834 INFO:teuthology.orchestra.run.smithi191.stdout:(91/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 509 kB/s | 25 kB 00:00 2024-03-28T14:59:15.122 INFO:teuthology.orchestra.run.smithi186.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 501 kB/s | 309 kB 00:00 2024-03-28T14:59:15.147 INFO:teuthology.orchestra.run.smithi186.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 783 kB/s | 548 kB 00:00 2024-03-28T14:59:15.231 INFO:teuthology.orchestra.run.smithi186.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-03-28T14:59:15.339 INFO:teuthology.orchestra.run.smithi186.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-03-28T14:59:15.401 INFO:teuthology.orchestra.run.smithi191.stdout:(92/174): ceph-test-19.0.0-2499.ge0e48ba6.el9.x 5.0 MB/s | 48 MB 00:09 2024-03-28T14:59:15.448 INFO:teuthology.orchestra.run.smithi186.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-28T14:59:15.485 INFO:teuthology.orchestra.run.smithi191.stdout:(93/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 6.4 MB/s | 4.1 MB 00:00 2024-03-28T14:59:15.540 INFO:teuthology.orchestra.run.smithi186.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-28T14:59:15.543 INFO:teuthology.orchestra.run.smithi191.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 15 MB/s | 911 kB 00:00 2024-03-28T14:59:15.590 INFO:teuthology.orchestra.run.smithi186.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 475 kB/s | 67 kB 00:00 2024-03-28T14:59:15.594 INFO:teuthology.orchestra.run.smithi191.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-03-28T14:59:15.703 INFO:teuthology.orchestra.run.smithi191.stdout:(96/174): postgresql-13.14-1.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-03-28T14:59:15.786 INFO:teuthology.orchestra.run.smithi191.stdout:(97/174): postgresql-server-13.14-1.el9.x86_64. 30 MB/s | 5.8 MB 00:00 2024-03-28T14:59:15.812 INFO:teuthology.orchestra.run.smithi191.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-28T14:59:15.846 INFO:teuthology.orchestra.run.smithi191.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.1 MB/s | 248 kB 00:00 2024-03-28T14:59:15.871 INFO:teuthology.orchestra.run.smithi191.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-03-28T14:59:15.897 INFO:teuthology.orchestra.run.smithi191.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 711 kB/s | 36 kB 00:00 2024-03-28T14:59:15.922 INFO:teuthology.orchestra.run.smithi191.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 942 kB/s | 47 kB 00:00 2024-03-28T14:59:15.947 INFO:teuthology.orchestra.run.smithi191.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-28T14:59:16.016 INFO:teuthology.orchestra.run.smithi186.stdout:(113/174): ceph-test-19.0.0-2499.ge0e48ba6.el9. 3.4 MB/s | 48 MB 00:13 2024-03-28T14:59:16.041 INFO:teuthology.orchestra.run.smithi186.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 302 kB/s | 151 kB 00:00 2024-03-28T14:59:16.116 INFO:teuthology.orchestra.run.smithi186.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.6 MB/s | 838 kB 00:00 2024-03-28T14:59:16.158 INFO:teuthology.orchestra.run.smithi186.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 372 kB/s | 43 kB 00:00 2024-03-28T14:59:16.198 INFO:teuthology.orchestra.run.smithi191.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 81 kB/s | 22 kB 00:00 2024-03-28T14:59:16.242 INFO:teuthology.orchestra.run.smithi186.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 328 kB/s | 32 kB 00:00 2024-03-28T14:59:16.267 INFO:teuthology.orchestra.run.smithi186.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 137 kB/s | 15 kB 00:00 2024-03-28T14:59:16.334 INFO:teuthology.orchestra.run.smithi186.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-28T14:59:16.348 INFO:teuthology.orchestra.run.smithi191.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-03-28T14:59:16.393 INFO:teuthology.orchestra.run.smithi186.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-03-28T14:59:16.434 INFO:teuthology.orchestra.run.smithi186.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-03-28T14:59:16.493 INFO:teuthology.orchestra.run.smithi186.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-03-28T14:59:16.535 INFO:teuthology.orchestra.run.smithi186.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-28T14:59:16.557 INFO:teuthology.orchestra.run.smithi191.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-03-28T14:59:16.624 INFO:teuthology.orchestra.run.smithi191.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-03-28T14:59:16.635 INFO:teuthology.orchestra.run.smithi186.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-28T14:59:16.650 INFO:teuthology.orchestra.run.smithi191.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-03-28T14:59:16.708 INFO:teuthology.orchestra.run.smithi191.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-03-28T14:59:16.727 INFO:teuthology.orchestra.run.smithi186.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 117 kB/s | 11 kB 00:00 2024-03-28T14:59:16.752 INFO:teuthology.orchestra.run.smithi186.stdout:(126/174): python3-asyncssh-2.13.2-5.el9.noarch 744 kB/s | 548 kB 00:00 2024-03-28T14:59:16.784 INFO:teuthology.orchestra.run.smithi191.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-28T14:59:16.794 INFO:teuthology.orchestra.run.smithi186.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.8 MB/s | 2.0 MB 00:00 2024-03-28T14:59:16.820 INFO:teuthology.orchestra.run.smithi186.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-03-28T14:59:16.845 INFO:teuthology.orchestra.run.smithi186.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-03-28T14:59:16.887 INFO:teuthology.orchestra.run.smithi186.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-03-28T14:59:16.912 INFO:teuthology.orchestra.run.smithi186.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-03-28T14:59:16.917 INFO:teuthology.orchestra.run.smithi191.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-28T14:59:16.943 INFO:teuthology.orchestra.run.smithi191.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 424 kB/s | 67 kB 00:00 2024-03-28T14:59:16.979 INFO:teuthology.orchestra.run.smithi186.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-03-28T14:59:17.010 INFO:teuthology.orchestra.run.smithi191.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-28T14:59:17.013 INFO:teuthology.orchestra.run.smithi186.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 787 kB/s | 79 kB 00:00 2024-03-28T14:59:17.072 INFO:teuthology.orchestra.run.smithi186.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 4.5 MB/s | 1.0 MB 00:00 2024-03-28T14:59:17.088 INFO:teuthology.orchestra.run.smithi186.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 530 kB/s | 58 kB 00:00 2024-03-28T14:59:17.114 INFO:teuthology.orchestra.run.smithi186.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-03-28T14:59:17.164 INFO:teuthology.orchestra.run.smithi186.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 178 kB/s | 16 kB 00:00 2024-03-28T14:59:17.189 INFO:teuthology.orchestra.run.smithi186.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 895 kB/s | 90 kB 00:00 2024-03-28T14:59:17.206 INFO:teuthology.orchestra.run.smithi186.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-03-28T14:59:17.229 INFO:teuthology.orchestra.run.smithi191.stdout:(114/174): ceph-osd-19.0.0-2499.ge0e48ba6.el9.x 1.4 MB/s | 17 MB 00:11 2024-03-28T14:59:17.256 INFO:teuthology.orchestra.run.smithi186.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-03-28T14:59:17.269 INFO:teuthology.orchestra.run.smithi191.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-03-28T14:59:17.290 INFO:teuthology.orchestra.run.smithi186.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-28T14:59:17.295 INFO:teuthology.orchestra.run.smithi191.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 1.9 MB/s | 548 kB 00:00 2024-03-28T14:59:17.315 INFO:teuthology.orchestra.run.smithi186.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 542 kB/s | 59 kB 00:00 2024-03-28T14:59:17.337 INFO:teuthology.orchestra.run.smithi191.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 480 kB/s | 32 kB 00:00 2024-03-28T14:59:17.357 INFO:teuthology.orchestra.run.smithi186.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 355 kB/s | 36 kB 00:00 2024-03-28T14:59:17.370 INFO:teuthology.orchestra.run.smithi191.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 198 kB/s | 15 kB 00:00 2024-03-28T14:59:17.391 INFO:teuthology.orchestra.run.smithi186.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-03-28T14:59:17.412 INFO:teuthology.orchestra.run.smithi191.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-28T14:59:17.416 INFO:teuthology.orchestra.run.smithi186.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-28T14:59:17.450 INFO:teuthology.orchestra.run.smithi186.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 969 kB/s | 90 kB 00:00 2024-03-28T14:59:17.455 INFO:teuthology.orchestra.run.smithi191.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-28T14:59:17.480 INFO:teuthology.orchestra.run.smithi191.stdout:(121/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 172 kB/s | 43 kB 00:00 2024-03-28T14:59:17.500 INFO:teuthology.orchestra.run.smithi186.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-03-28T14:59:17.505 INFO:teuthology.orchestra.run.smithi191.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-28T14:59:17.526 INFO:teuthology.orchestra.run.smithi186.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 183 kB/s | 20 kB 00:00 2024-03-28T14:59:17.539 INFO:teuthology.orchestra.run.smithi191.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 419 kB/s | 35 kB 00:00 2024-03-28T14:59:17.559 INFO:teuthology.orchestra.run.smithi186.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-28T14:59:17.640 INFO:teuthology.orchestra.run.smithi191.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-28T14:59:17.642 INFO:teuthology.orchestra.run.smithi186.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-28T14:59:17.706 INFO:teuthology.orchestra.run.smithi191.stdout:(125/174): python3-google-auth-2.28.2-1.el9.noa 896 kB/s | 203 kB 00:00 2024-03-28T14:59:17.709 INFO:teuthology.orchestra.run.smithi186.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.7 MB/s | 321 kB 00:00 2024-03-28T14:59:17.732 INFO:teuthology.orchestra.run.smithi191.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-03-28T14:59:17.791 INFO:teuthology.orchestra.run.smithi191.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 210 kB/s | 18 kB 00:00 2024-03-28T14:59:17.824 INFO:teuthology.orchestra.run.smithi191.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-03-28T14:59:17.875 INFO:teuthology.orchestra.run.smithi191.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-03-28T14:59:17.934 INFO:teuthology.orchestra.run.smithi191.stdout:(130/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.8 MB/s | 2.0 MB 00:00 2024-03-28T14:59:17.959 INFO:teuthology.orchestra.run.smithi191.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 147 kB/s | 20 kB 00:00 2024-03-28T14:59:17.968 INFO:teuthology.orchestra.run.smithi186.stdout:(152/174): librbd1-19.0.0-2499.ge0e48ba6.el9.x8 7.7 MB/s | 3.1 MB 00:00 2024-03-28T14:59:17.993 INFO:teuthology.orchestra.run.smithi191.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-03-28T14:59:18.018 INFO:teuthology.orchestra.run.smithi191.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 549 kB/s | 46 kB 00:00 2024-03-28T14:59:18.043 INFO:teuthology.orchestra.run.smithi191.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 935 kB/s | 79 kB 00:00 2024-03-28T14:59:18.069 INFO:teuthology.orchestra.run.smithi191.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 759 kB/s | 58 kB 00:00 2024-03-28T14:59:18.120 INFO:teuthology.orchestra.run.smithi191.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-03-28T14:59:18.145 INFO:teuthology.orchestra.run.smithi191.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-03-28T14:59:18.170 INFO:teuthology.orchestra.run.smithi191.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 891 kB/s | 90 kB 00:00 2024-03-28T14:59:18.196 INFO:teuthology.orchestra.run.smithi191.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 372 kB/s | 28 kB 00:00 2024-03-28T14:59:18.221 INFO:teuthology.orchestra.run.smithi191.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 405 kB/s | 31 kB 00:00 2024-03-28T14:59:18.255 INFO:teuthology.orchestra.run.smithi191.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-28T14:59:18.280 INFO:teuthology.orchestra.run.smithi191.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 702 kB/s | 59 kB 00:00 2024-03-28T14:59:18.293 INFO:teuthology.orchestra.run.smithi186.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 4.0 MB/s | 2.3 MB 00:00 2024-03-28T14:59:18.305 INFO:teuthology.orchestra.run.smithi191.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 425 kB/s | 36 kB 00:00 2024-03-28T14:59:18.339 INFO:teuthology.orchestra.run.smithi191.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 606 kB/s | 51 kB 00:00 2024-03-28T14:59:18.364 INFO:teuthology.orchestra.run.smithi191.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-28T14:59:18.423 INFO:teuthology.orchestra.run.smithi191.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-28T14:59:18.448 INFO:teuthology.orchestra.run.smithi191.stdout:(147/174): python3-websocket-client-1.2.3-2.el9 627 kB/s | 90 kB 00:00 2024-03-28T14:59:18.466 INFO:teuthology.orchestra.run.smithi191.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 197 kB/s | 20 kB 00:00 2024-03-28T14:59:18.486 INFO:teuthology.orchestra.run.smithi186.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.5 MB/s | 1.8 MB 00:00 2024-03-28T14:59:18.499 INFO:teuthology.orchestra.run.smithi191.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-28T14:59:18.503 INFO:teuthology.orchestra.run.smithi186.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 202 kB/s | 42 kB 00:00 2024-03-28T14:59:18.570 INFO:teuthology.orchestra.run.smithi186.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-28T14:59:18.620 INFO:teuthology.orchestra.run.smithi186.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 730 kB/s | 36 kB 00:00 2024-03-28T14:59:18.653 INFO:teuthology.orchestra.run.smithi186.stdout:(158/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.5 MB/s | 248 kB 00:00 2024-03-28T14:59:18.683 INFO:teuthology.orchestra.run.smithi191.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.7 MB/s | 1.6 MB 00:00 2024-03-28T14:59:18.762 INFO:teuthology.orchestra.run.smithi186.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-28T14:59:18.804 INFO:teuthology.orchestra.run.smithi186.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.0 MB/s | 192 kB 00:00 2024-03-28T14:59:18.809 INFO:teuthology.orchestra.run.smithi191.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.5 MB/s | 321 kB 00:00 2024-03-28T14:59:18.821 INFO:teuthology.orchestra.run.smithi186.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 392 kB/s | 23 kB 00:00 2024-03-28T14:59:18.879 INFO:teuthology.orchestra.run.smithi186.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-28T14:59:18.913 INFO:teuthology.orchestra.run.smithi186.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-28T14:59:18.963 INFO:teuthology.orchestra.run.smithi186.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 136 kB 00:00 2024-03-28T14:59:18.980 INFO:teuthology.orchestra.run.smithi186.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 955 kB/s | 64 kB 00:00 2024-03-28T14:59:19.014 INFO:teuthology.orchestra.run.smithi186.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 572 kB/s | 29 kB 00:00 2024-03-28T14:59:19.026 INFO:teuthology.orchestra.run.smithi191.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-03-28T14:59:19.093 INFO:teuthology.orchestra.run.smithi191.stdout:(153/174): librbd1-19.0.0-2499.ge0e48ba6.el9.x8 5.3 MB/s | 3.1 MB 00:00 2024-03-28T14:59:19.143 INFO:teuthology.orchestra.run.smithi191.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 16 MB/s | 1.8 MB 00:00 2024-03-28T14:59:19.189 INFO:teuthology.orchestra.run.smithi186.stdout:(167/174): librados2-19.0.0-2499.ge0e48ba6.el9. 2.0 MB/s | 3.4 MB 00:01 2024-03-28T14:59:19.193 INFO:teuthology.orchestra.run.smithi191.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.9 MB/s | 248 kB 00:00 2024-03-28T14:59:19.227 INFO:teuthology.orchestra.run.smithi191.stdout:(156/174): NetworkManager-team-1.47.2-1.el9.x86 314 kB/s | 42 kB 00:00 2024-03-28T14:59:19.253 INFO:teuthology.orchestra.run.smithi191.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-28T14:59:19.265 INFO:teuthology.orchestra.run.smithi186.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.3 MB/s | 1.2 MB 00:00 2024-03-28T14:59:19.295 INFO:teuthology.orchestra.run.smithi191.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 543 kB/s | 36 kB 00:00 2024-03-28T14:59:19.320 INFO:teuthology.orchestra.run.smithi191.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-28T14:59:19.362 INFO:teuthology.orchestra.run.smithi191.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 553 kB/s | 23 kB 00:00 2024-03-28T14:59:19.396 INFO:teuthology.orchestra.run.smithi191.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.5 MB/s | 256 kB 00:00 2024-03-28T14:59:19.421 INFO:teuthology.orchestra.run.smithi191.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-03-28T14:59:19.447 INFO:teuthology.orchestra.run.smithi191.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-28T14:59:19.448 INFO:teuthology.orchestra.run.smithi186.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.0 MB/s | 2.2 MB 00:00 2024-03-28T14:59:19.472 INFO:teuthology.orchestra.run.smithi191.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-28T14:59:19.482 INFO:teuthology.orchestra.run.smithi186.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 2.1 MB/s | 465 kB 00:00 2024-03-28T14:59:19.497 INFO:teuthology.orchestra.run.smithi191.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-28T14:59:19.515 INFO:teuthology.orchestra.run.smithi186.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 465 kB/s | 31 kB 00:00 2024-03-28T14:59:19.523 INFO:teuthology.orchestra.run.smithi191.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 570 kB/s | 29 kB 00:00 2024-03-28T14:59:19.598 INFO:teuthology.orchestra.run.smithi191.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 29 MB/s | 2.2 MB 00:00 2024-03-28T14:59:19.641 INFO:teuthology.orchestra.run.smithi191.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 8.5 MB/s | 1.2 MB 00:00 2024-03-28T14:59:19.682 INFO:teuthology.orchestra.run.smithi186.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.6 MB/s | 2.3 MB 00:00 2024-03-28T14:59:19.699 INFO:teuthology.orchestra.run.smithi191.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 23 MB/s | 2.3 MB 00:00 2024-03-28T14:59:19.741 INFO:teuthology.orchestra.run.smithi186.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 334 kB/s | 74 kB 00:00 2024-03-28T14:59:19.758 INFO:teuthology.orchestra.run.smithi191.stdout:(170/174): librados2-19.0.0-2499.ge0e48ba6.el9. 2.6 MB/s | 3.4 MB 00:01 2024-03-28T14:59:19.775 INFO:teuthology.orchestra.run.smithi191.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 412 kB/s | 31 kB 00:00 2024-03-28T14:59:19.809 INFO:teuthology.orchestra.run.smithi191.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 2.7 MB/s | 465 kB 00:00 2024-03-28T14:59:19.816 INFO:teuthology.orchestra.run.smithi186.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 3.2 MB/s | 1.1 MB 00:00 2024-03-28T14:59:19.822 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-28T14:59:19.822 INFO:teuthology.orchestra.run.smithi186.stdout:Total 10 MB/s | 227 MB 00:21 2024-03-28T14:59:19.909 INFO:teuthology.orchestra.run.smithi191.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 556 kB/s | 74 kB 00:00 2024-03-28T14:59:20.018 INFO:teuthology.orchestra.run.smithi191.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.1 MB/s | 1.1 MB 00:00 2024-03-28T14:59:20.023 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-28T14:59:20.023 INFO:teuthology.orchestra.run.smithi191.stdout:Total 12 MB/s | 227 MB 00:18 2024-03-28T14:59:20.402 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-28T14:59:20.595 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-28T14:59:20.683 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-28T14:59:20.684 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-28T14:59:20.872 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-28T14:59:20.872 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-28T14:59:22.750 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-28T14:59:22.750 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-28T14:59:22.922 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-28T14:59:22.923 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-28T14:59:29.970 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-28T14:59:30.006 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-28T14:59:30.087 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T14:59:30.134 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T14:59:30.176 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T14:59:30.213 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T14:59:30.223 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T14:59:30.273 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T14:59:30.305 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T14:59:30.334 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T14:59:30.360 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T14:59:30.367 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T14:59:30.371 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T14:59:30.392 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T14:59:30.420 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T14:59:30.440 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T14:59:30.479 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T14:59:30.521 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T14:59:30.563 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T14:59:30.613 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T14:59:30.765 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T14:59:30.810 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 10/198 2024-03-28T14:59:30.931 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 10/198 2024-03-28T14:59:30.945 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T14:59:30.961 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 11/198 2024-03-28T14:59:30.997 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 10/198 2024-03-28T14:59:31.052 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 11/198 2024-03-28T14:59:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:19.0.0-2499.ge0e48ba6.el9.x86_64 12/198 2024-03-28T14:59:31.101 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 10/198 2024-03-28T14:59:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T14:59:31.131 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 11/198 2024-03-28T14:59:31.235 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 11/198 2024-03-28T14:59:31.261 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-2499.ge0e48ba6.el9.x86_64 12/198 2024-03-28T14:59:31.298 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T14:59:31.319 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T14:59:31.372 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T14:59:31.400 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:19.0.0-2499.ge0e48ba6.el9.x86_64 16/198 2024-03-28T14:59:31.477 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T14:59:31.505 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2499.ge0e48ba6.el9.x86_64 16/198 2024-03-28T14:59:31.534 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T14:59:31.534 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:19.0.0-2499.ge0e48ba6.el9.x86 17/198 2024-03-28T14:59:31.562 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-2499.ge0e48ba6.el9.x86_64 16/198 2024-03-28T14:59:31.611 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2499.ge0e48ba6.el9.x86 17/198 2024-03-28T14:59:31.647 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T14:59:31.665 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2499.ge0e48ba6.el9.x86_64 16/198 2024-03-28T14:59:31.678 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T14:59:31.694 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T14:59:31.709 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-2499.ge0e48ba6.el9.x86 17/198 2024-03-28T14:59:31.770 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2499.ge0e48ba6.el9.x86 17/198 2024-03-28T14:59:31.809 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T14:59:31.831 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T14:59:31.846 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T14:59:31.847 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T14:59:31.879 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T14:59:31.924 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T14:59:32.039 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T14:59:32.071 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T14:59:32.099 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T14:59:32.120 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T14:59:32.285 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T14:59:32.306 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T14:59:32.457 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T14:59:32.509 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T14:59:32.595 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T14:59:32.609 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T14:59:32.660 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T14:59:32.693 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T14:59:32.779 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T14:59:32.897 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T14:59:33.013 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T14:59:33.015 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T14:59:33.108 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T14:59:33.116 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T14:59:33.140 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T14:59:33.167 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T14:59:33.202 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T14:59:33.204 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T14:59:33.234 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T14:59:33.261 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T14:59:33.275 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T14:59:33.313 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T14:59:33.329 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T14:59:33.368 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T14:59:33.386 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T14:59:33.434 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T14:59:33.441 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T14:59:33.467 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T14:59:33.479 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T14:59:33.510 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T14:59:33.542 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T14:59:33.544 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T14:59:33.571 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T14:59:33.581 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T14:59:33.630 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T14:59:33.640 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T14:59:33.672 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T14:59:33.676 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-2499.ge0e48ba6.el9. 39/198 2024-03-28T14:59:33.707 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T14:59:33.719 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-2499.ge0e48ba6.el 40/198 2024-03-28T14:59:33.748 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-2499.ge0e48ba6.el9.x86_6 41/198 2024-03-28T14:59:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T14:59:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:19.0.0-2499.ge0e48ba6.el9. 39/198 2024-03-28T14:59:33.825 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T14:59:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:19.0.0-2499.ge0e48ba6.el 40/198 2024-03-28T14:59:33.871 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T14:59:33.892 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:19.0.0-2499.ge0e48ba6.el9.x86_6 41/198 2024-03-28T14:59:33.945 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T14:59:33.961 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T14:59:34.006 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T14:59:34.077 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T14:59:34.081 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T14:59:34.132 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T14:59:34.222 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T14:59:34.275 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T14:59:34.650 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T14:59:34.726 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T14:59:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T14:59:34.804 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T14:59:34.858 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T14:59:34.870 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T14:59:34.872 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T14:59:34.904 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T14:59:34.955 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T14:59:34.971 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T14:59:34.993 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_6 53/198 2024-03-28T14:59:35.024 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T14:59:35.024 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T14:59:35.034 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T14:59:35.065 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T14:59:35.114 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T14:59:35.151 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_6 53/198 2024-03-28T14:59:35.182 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T14:59:35.291 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T14:59:35.475 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T14:59:35.567 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T14:59:35.611 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T14:59:35.702 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T14:59:35.751 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T14:59:35.831 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T14:59:35.887 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T14:59:35.925 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T14:59:35.959 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T14:59:35.990 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T14:59:36.008 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T14:59:36.027 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T14:59:36.060 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T14:59:36.073 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T14:59:36.109 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T14:59:36.161 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T14:59:36.202 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T14:59:36.214 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T14:59:36.272 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T14:59:36.335 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T14:59:36.336 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T14:59:36.375 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T14:59:36.406 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T14:59:36.408 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T14:59:36.439 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T14:59:36.447 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T14:59:36.476 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T14:59:36.531 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T14:59:36.540 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T14:59:36.575 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T14:59:36.581 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T14:59:36.612 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T14:59:36.618 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T14:59:36.646 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T14:59:36.661 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T14:59:36.671 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T14:59:36.751 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T14:59:36.795 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T14:59:36.841 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T14:59:36.871 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T14:59:36.880 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T14:59:36.913 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T14:59:36.988 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T14:59:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T14:59:37.029 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T14:59:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T14:59:37.077 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T14:59:37.103 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T14:59:37.128 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T14:59:37.134 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T14:59:37.153 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T14:59:37.235 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T14:59:37.325 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T14:59:37.362 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T14:59:37.375 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T14:59:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T14:59:37.503 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T14:59:37.511 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T14:59:37.540 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T14:59:37.560 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T14:59:37.573 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T14:59:37.611 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T14:59:37.616 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T14:59:37.640 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T14:59:37.674 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T14:59:37.713 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T14:59:37.745 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T14:59:37.775 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T14:59:37.799 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T14:59:37.849 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T14:59:37.849 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T14:59:37.885 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T14:59:37.919 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T14:59:37.959 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T14:59:37.985 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T14:59:37.994 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T14:59:38.030 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T14:59:38.062 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T14:59:38.082 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T14:59:38.093 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T14:59:38.121 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T14:59:38.155 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T14:59:38.172 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T14:59:38.194 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T14:59:38.217 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T14:59:38.234 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T14:59:38.248 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T14:59:38.264 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T14:59:38.295 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T14:59:38.340 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T14:59:38.375 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T14:59:38.391 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T14:59:38.408 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T14:59:38.449 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T14:59:38.450 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T14:59:38.484 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T14:59:38.572 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T14:59:38.662 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T14:59:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T14:59:38.746 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T14:59:38.793 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T14:59:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T14:59:38.939 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T14:59:39.182 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T14:59:39.231 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T14:59:39.268 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T14:59:39.316 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T14:59:39.398 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T14:59:39.436 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T14:59:39.613 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T14:59:39.642 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T14:59:39.670 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T14:59:39.719 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T14:59:39.737 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T14:59:39.770 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T14:59:39.818 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T14:59:39.898 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T14:59:39.929 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T14:59:40.108 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T14:59:40.144 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T14:59:40.230 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T14:59:40.387 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T14:59:40.497 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T14:59:40.908 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T14:59:41.016 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T14:59:41.719 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T14:59:41.801 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T14:59:41.832 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T14:59:41.863 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T14:59:41.891 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T14:59:42.205 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T14:59:42.225 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T14:59:42.229 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-2499.ge0e48ba6.el9.x86_64 124/198 2024-03-28T14:59:42.303 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T14:59:42.320 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-2499.ge0e48ba6.el9.x86_64 124/198 2024-03-28T14:59:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T14:59:42.351 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 125/198 2024-03-28T14:59:42.365 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T14:59:42.387 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T14:59:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T14:59:42.712 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T14:59:42.739 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:19.0.0-2499.ge0e48ba6.el9.x86_64 124/198 2024-03-28T14:59:42.829 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:19.0.0-2499.ge0e48ba6.el9.x86_64 124/198 2024-03-28T14:59:42.847 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T14:59:42.883 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 128/198 2024-03-28T14:59:42.885 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 125/198 2024-03-28T14:59:42.922 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T14:59:42.953 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 128/198 2024-03-28T14:59:42.976 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 129/198 2024-03-28T14:59:43.380 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T14:59:43.417 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 128/198 2024-03-28T14:59:43.480 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 128/198 2024-03-28T14:59:43.518 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 129/198 2024-03-28T14:59:45.122 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 130/198 2024-03-28T14:59:45.156 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 130/198 2024-03-28T14:59:45.702 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 130/198 2024-03-28T14:59:45.764 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 130/198 2024-03-28T14:59:45.767 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-2499.ge0e48ba6.el9.x86_64 131/198 2024-03-28T14:59:45.798 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 130/198 2024-03-28T14:59:45.852 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-2499.ge0e48ba6.el9.x86_64 131/198 2024-03-28T14:59:45.853 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T14:59:45.853 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T14:59:45.853 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:59:45.877 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-2499.ge0e48ba6.el9.x86_64 132/198 2024-03-28T14:59:46.337 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 130/198 2024-03-28T14:59:46.393 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:19.0.0-2499.ge0e48ba6.el9.x86_64 131/198 2024-03-28T14:59:46.493 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:19.0.0-2499.ge0e48ba6.el9.x86_64 131/198 2024-03-28T14:59:46.494 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T14:59:46.494 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T14:59:46.494 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:59:46.536 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:19.0.0-2499.ge0e48ba6.el9.x86_64 132/198 2024-03-28T14:59:57.922 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2499.ge0e48ba6.el9.x86_64 132/198 2024-03-28T14:59:57.922 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-03-28T14:59:57.922 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-03-28T14:59:57.922 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-03-28T14:59:57.922 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-03-28T14:59:57.922 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-03-28T14:59:57.922 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-03-28T14:59:57.923 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-03-28T14:59:57.923 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:59:58.093 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-2499.ge0e48ba6.el9.noarch 133/198 2024-03-28T14:59:58.109 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-2499.ge0e48ba6.el9.noarch 133/198 2024-03-28T14:59:58.109 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T14:59:58.109 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T14:59:58.109 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:59:58.380 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2499.ge0e48ba6.el9.x86_64 132/198 2024-03-28T14:59:58.380 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-03-28T14:59:58.380 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-03-28T14:59:58.380 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-03-28T14:59:58.380 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-03-28T14:59:58.381 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-03-28T14:59:58.381 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-03-28T14:59:58.381 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-03-28T14:59:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:59:58.542 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:19.0.0-2499.ge0e48ba6.el9.noarch 133/198 2024-03-28T14:59:58.561 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:19.0.0-2499.ge0e48ba6.el9.noarch 133/198 2024-03-28T14:59:58.561 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T14:59:58.561 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T14:59:58.561 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T14:59:59.579 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 134/198 2024-03-28T14:59:59.605 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 134/198 2024-03-28T14:59:59.605 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T14:59:59.605 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T14:59:59.605 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T14:59:59.605 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T14:59:59.605 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T14:59:59.831 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-2499.ge0e48ba6.el9.x86_64 135/198 2024-03-28T14:59:59.854 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-2499.ge0e48ba6.el9.x86_64 135/198 2024-03-28T14:59:59.854 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T14:59:59.854 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T14:59:59.854 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T14:59:59.854 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T14:59:59.854 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 134/198 2024-03-28T15:00:00.078 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 134/198 2024-03-28T15:00:00.078 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:00.078 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T15:00:00.079 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:00:00.079 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:00:00.079 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:00.297 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-2499.ge0e48ba6.el9.x86_64 136/198 2024-03-28T15:00:00.304 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:19.0.0-2499.ge0e48ba6.el9.x86_64 135/198 2024-03-28T15:00:00.327 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:19.0.0-2499.ge0e48ba6.el9.x86_64 135/198 2024-03-28T15:00:00.327 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:00.327 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T15:00:00.327 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:00:00.327 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:00:00.327 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:00.335 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-2499.ge0e48ba6.el9.x86_64 136/198 2024-03-28T15:00:00.335 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:00.335 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T15:00:00.335 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:00:00.336 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:00:00.336 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:00.441 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T15:00:00.764 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:19.0.0-2499.ge0e48ba6.el9.x86_64 136/198 2024-03-28T15:00:00.786 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:19.0.0-2499.ge0e48ba6.el9.x86_64 136/198 2024-03-28T15:00:00.787 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:00.787 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T15:00:00.787 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:00:00.787 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:00:00.787 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:00.834 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-2499.ge0e48ba6.el9.noarch 138/198 2024-03-28T15:00:00.866 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-2499.ge0e48ba6.el9.noarch 138/198 2024-03-28T15:00:00.882 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T15:00:00.896 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2499.ge0e48ba6.e 139/198 2024-03-28T15:00:00.957 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2499.ge0e48ba6. 140/198 2024-03-28T15:00:00.987 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2499.ge0e48ba6.el9.noa 141/198 2024-03-28T15:00:01.341 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.0.0-2499.ge0e48ba6.el9.noarch 138/198 2024-03-28T15:00:01.372 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:19.0.0-2499.ge0e48ba6.el9.noarch 138/198 2024-03-28T15:00:01.404 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2499.ge0e48ba6.e 139/198 2024-03-28T15:00:01.465 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2499.ge0e48ba6. 140/198 2024-03-28T15:00:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2499.ge0e48ba6.el9.noa 141/198 2024-03-28T15:00:02.024 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2499.ge0e48ba6.el9.noa 141/198 2024-03-28T15:00:02.056 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2499.ge0e48ba6.el9.n 142/198 2024-03-28T15:00:02.548 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2499.ge0e48ba6.el9.noa 141/198 2024-03-28T15:00:02.589 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2499.ge0e48ba6.el9.n 142/198 2024-03-28T15:00:03.020 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2499.ge0e48ba6.el9.n 142/198 2024-03-28T15:00:03.048 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2499.ge0e 143/198 2024-03-28T15:00:03.148 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2499.ge0e 143/198 2024-03-28T15:00:03.312 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2499.ge0e48ba6.el 144/198 2024-03-28T15:00:03.350 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-2499.ge0e48ba6.el9.x86_64 145/198 2024-03-28T15:00:03.373 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2499.ge0e48ba6.el9.x86_64 145/198 2024-03-28T15:00:03.373 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:03.373 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T15:00:03.373 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:00:03.373 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:00:03.373 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:03.418 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-2499.ge0e48ba6.el9.noarch 146/198 2024-03-28T15:00:03.429 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2499.ge0e48ba6.el9.noarch 146/198 2024-03-28T15:00:03.489 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-2499.ge0e48ba6.el9.x86_64 147/198 2024-03-28T15:00:03.553 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2499.ge0e48ba6.el9.n 142/198 2024-03-28T15:00:03.580 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2499.ge0e 143/198 2024-03-28T15:00:03.680 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2499.ge0e 143/198 2024-03-28T15:00:03.844 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2499.ge0e48ba6.el 144/198 2024-03-28T15:00:03.874 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:19.0.0-2499.ge0e48ba6.el9.x86_64 145/198 2024-03-28T15:00:03.897 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2499.ge0e48ba6.el9.x86_64 145/198 2024-03-28T15:00:03.897 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:03.897 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T15:00:03.897 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:00:03.897 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:00:03.898 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:03.941 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:19.0.0-2499.ge0e48ba6.el9.noarch 146/198 2024-03-28T15:00:03.952 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2499.ge0e48ba6.el9.noarch 146/198 2024-03-28T15:00:04.000 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:19.0.0-2499.ge0e48ba6.el9.x86_64 147/198 2024-03-28T15:00:04.370 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T15:00:04.403 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 149/198 2024-03-28T15:00:04.426 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 149/198 2024-03-28T15:00:04.426 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:04.426 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T15:00:04.426 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:00:04.427 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:00:04.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:04.471 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2499.ge0e48 150/198 2024-03-28T15:00:04.493 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2499.ge0e48 150/198 2024-03-28T15:00:04.493 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:04.493 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T15:00:04.493 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:04.546 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 151/198 2024-03-28T15:00:04.569 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 151/198 2024-03-28T15:00:04.569 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:04.569 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T15:00:04.569 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:00:04.569 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:00:04.569 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:04.872 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 152/198 2024-03-28T15:00:04.895 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 152/198 2024-03-28T15:00:04.895 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:04.895 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T15:00:04.895 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:00:04.895 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:00:04.895 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:04.915 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T15:00:04.952 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 149/198 2024-03-28T15:00:04.975 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 149/198 2024-03-28T15:00:04.975 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:04.975 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T15:00:04.976 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:00:04.976 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:00:04.976 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:05.019 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2499.ge0e48 150/198 2024-03-28T15:00:05.041 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2499.ge0e48 150/198 2024-03-28T15:00:05.041 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:05.041 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T15:00:05.041 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 151/198 2024-03-28T15:00:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 151/198 2024-03-28T15:00:05.109 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:05.109 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T15:00:05.110 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:00:05.110 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:00:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:05.412 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 152/198 2024-03-28T15:00:05.435 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 152/198 2024-03-28T15:00:05.435 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:00:05.435 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T15:00:05.435 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:00:05.435 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:00:05.435 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:09.368 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-2499.ge0e48ba6.el9.x86_64 153/198 2024-03-28T15:00:09.410 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 154/198 2024-03-28T15:00:09.440 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 155/198 2024-03-28T15:00:09.467 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T15:00:09.649 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-shell-2:19.0.0-2499.ge0e48ba6.el9.x86_64 157/198 2024-03-28T15:00:09.758 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T15:00:09.806 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 159/198 2024-03-28T15:00:09.857 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T15:00:09.908 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:19.0.0-2499.ge0e48ba6.el9.x86_64 153/198 2024-03-28T15:00:09.923 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T15:00:09.950 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 154/198 2024-03-28T15:00:09.958 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T15:00:09.988 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 155/198 2024-03-28T15:00:09.991 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_ 163/198 2024-03-28T15:00:10.015 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T15:00:10.023 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T15:00:10.053 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T15:00:10.119 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T15:00:10.177 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T15:00:10.219 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T15:00:10.221 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-shell-2:19.0.0-2499.ge0e48ba6.el9.x86_64 157/198 2024-03-28T15:00:10.333 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T15:00:10.379 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 159/198 2024-03-28T15:00:10.436 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T15:00:10.505 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T15:00:10.556 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T15:00:10.589 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_ 163/198 2024-03-28T15:00:10.621 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T15:00:10.651 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T15:00:10.727 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T15:00:10.805 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T15:00:10.844 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:00:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T15:00:10.891 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:00:10.911 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:00:11.477 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:00:11.522 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:00:11.541 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:00:11.743 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:19.0.0-2499.ge0e48ba6.el9.noarch 170/198 2024-03-28T15:00:11.822 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T15:00:11.864 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T15:00:11.945 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T15:00:11.976 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T15:00:12.001 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T15:00:12.026 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T15:00:12.053 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T15:00:12.053 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:00:12.171 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:00:12.212 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T15:00:12.212 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:00:12.297 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:00:12.332 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T15:00:12.357 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-top-2:19.0.0-2499.ge0e48ba6.el9.noarch 170/198 2024-03-28T15:00:12.365 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T15:00:12.390 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T15:00:12.415 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T15:00:12.434 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T15:00:12.449 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T15:00:12.482 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T15:00:12.487 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T15:00:12.520 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T15:00:12.545 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T15:00:12.570 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T15:00:12.574 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T15:00:12.596 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:00:12.612 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T15:00:12.621 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T15:00:12.628 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:00:12.628 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:00:12.637 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T15:00:12.662 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T15:00:12.689 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T15:00:12.690 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:00:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:00:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T15:00:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:00:12.824 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:00:12.824 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:00:12.899 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:00:12.899 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:00:12.923 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:00:12.926 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T15:00:12.956 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T15:00:12.959 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T15:00:12.983 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T15:00:12.984 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T15:00:13.009 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T15:00:13.016 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T15:00:13.044 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T15:00:13.082 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:00:13.099 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T15:00:13.130 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T15:00:13.156 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T15:00:13.181 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T15:00:13.206 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:00:13.231 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T15:00:13.281 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:00:13.282 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:00:13.434 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:00:13.434 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:00:13.517 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:00:13.542 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:00:13.574 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T15:00:13.602 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T15:00:13.627 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T15:00:13.692 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:00:18.399 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:00:18.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-2499.ge0e48ba6.el9.x86_64 1/198 2024-03-28T15:00:18.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2/198 2024-03-28T15:00:18.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 3/198 2024-03-28T15:00:18.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 4/198 2024-03-28T15:00:18.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2499.ge0e48 5/198 2024-03-28T15:00:18.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-2499.ge0e48ba6.el9.x86_64 6/198 2024-03-28T15:00:18.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-2499.ge0e48ba6.el9.x86_64 7/198 2024-03-28T15:00:18.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-2499.ge0e48ba6.el9.x86_64 8/198 2024-03-28T15:00:18.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 9/198 2024-03-28T15:00:18.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 10/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-2499.ge0e48ba6.el9.x86_64 11/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-2499.ge0e48ba6.el9.x86_64 12/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 13/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-shell-2:19.0.0-2499.ge0e48ba6.el9.x86_64 14/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_ 15/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 16/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-2499.ge0e48ba6.el9.x86_64 17/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_6 18/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-2499.ge0e48ba6.el9.x86 19/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 20/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2499.ge0e48ba6.el 21/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-2499.ge0e48ba6.el9. 22/198 2024-03-28T15:00:18.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-2499.ge0e48ba6.el9.x86_6 23/198 2024-03-28T15:00:18.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-2499.ge0e48ba6.el9.x86_64 24/198 2024-03-28T15:00:18.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 25/198 2024-03-28T15:00:18.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 26/198 2024-03-28T15:00:18.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 27/198 2024-03-28T15:00:18.419 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 28/198 2024-03-28T15:00:18.419 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 29/198 2024-03-28T15:00:18.419 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2499.ge0e48ba6. 30/198 2024-03-28T15:00:18.419 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2499.ge0e48ba6.el9.noa 31/198 2024-03-28T15:00:18.419 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2499.ge0e48ba6.el9.n 32/198 2024-03-28T15:00:18.419 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2499.ge0e 33/198 2024-03-28T15:00:18.419 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2499.ge0e48ba6.el 34/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2499.ge0e48ba6.el9.noarch 35/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2499.ge0e48ba6.e 36/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-2499.ge0e48ba6.el9.noarch 37/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-2499.ge0e48ba6.el9.noarch 38/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:19.0.0-2499.ge0e48ba6.el9.noarch 39/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T15:00:18.420 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T15:00:18.421 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T15:00:18.422 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T15:00:18.423 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T15:00:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T15:00:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T15:00:18.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T15:00:18.427 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 151/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-2499.ge0e48ba6.el9.x86_64 153/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T15:00:18.428 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T15:00:18.429 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T15:00:18.430 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T15:00:18.430 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T15:00:18.430 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T15:00:18.430 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T15:00:18.430 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T15:00:18.430 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T15:00:18.430 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T15:00:18.430 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T15:00:18.430 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T15:00:18.431 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T15:00:18.432 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:00:18.432 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T15:00:18.432 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T15:00:18.432 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T15:00:19.471 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:00:19.471 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:19.0.0-2499.ge0e48ba6.el9.x86_64 1/198 2024-03-28T15:00:19.471 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2/198 2024-03-28T15:00:19.471 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 3/198 2024-03-28T15:00:19.471 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 4/198 2024-03-28T15:00:19.471 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2499.ge0e48 5/198 2024-03-28T15:00:19.471 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:19.0.0-2499.ge0e48ba6.el9.x86_64 6/198 2024-03-28T15:00:19.471 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:19.0.0-2499.ge0e48ba6.el9.x86_64 7/198 2024-03-28T15:00:19.471 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:19.0.0-2499.ge0e48ba6.el9.x86_64 8/198 2024-03-28T15:00:19.471 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 9/198 2024-03-28T15:00:19.471 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 10/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:19.0.0-2499.ge0e48ba6.el9.x86_64 11/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:19.0.0-2499.ge0e48ba6.el9.x86_64 12/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 13/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-shell-2:19.0.0-2499.ge0e48ba6.el9.x86_64 14/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_ 15/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 16/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:19.0.0-2499.ge0e48ba6.el9.x86_64 17/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_6 18/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:19.0.0-2499.ge0e48ba6.el9.x86 19/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 20/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2499.ge0e48ba6.el 21/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:19.0.0-2499.ge0e48ba6.el9. 22/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:19.0.0-2499.ge0e48ba6.el9.x86_6 23/198 2024-03-28T15:00:19.472 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:19.0.0-2499.ge0e48ba6.el9.x86_64 24/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 25/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 26/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 27/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 28/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 29/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2499.ge0e48ba6. 30/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2499.ge0e48ba6.el9.noa 31/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2499.ge0e48ba6.el9.n 32/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2499.ge0e 33/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2499.ge0e48ba6.el 34/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2499.ge0e48ba6.el9.noarch 35/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2499.ge0e48ba6.e 36/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:19.0.0-2499.ge0e48ba6.el9.noarch 37/198 2024-03-28T15:00:19.473 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:19.0.0-2499.ge0e48ba6.el9.noarch 38/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-top-2:19.0.0-2499.ge0e48ba6.el9.noarch 39/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T15:00:19.474 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T15:00:19.475 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T15:00:19.476 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T15:00:19.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T15:00:19.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T15:00:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T15:00:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T15:00:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T15:00:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T15:00:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T15:00:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T15:00:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T15:00:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T15:00:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T15:00:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T15:00:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T15:00:19.481 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T15:00:19.481 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T15:00:19.481 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T15:00:19.481 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T15:00:19.481 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T15:00:19.481 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T15:00:19.481 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T15:00:19.481 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T15:00:19.481 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T15:00:19.481 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T15:00:19.481 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T15:00:19.482 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 151/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:19.0.0-2499.ge0e48ba6.el9.x86_64 153/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T15:00:19.483 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T15:00:19.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T15:00:19.485 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T15:00:19.486 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T15:00:19.487 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T15:00:21.313 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T15:00:21.313 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:21.313 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-03-28T15:00:21.313 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T15:00:21.313 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T15:00:21.313 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.314 INFO:teuthology.orchestra.run.smithi186.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T15:00:21.315 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T15:00:21.315 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T15:00:21.315 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T15:00:21.315 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T15:00:21.315 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T15:00:21.315 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T15:00:21.315 INFO:teuthology.orchestra.run.smithi186.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T15:00:21.315 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T15:00:21.315 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-28T15:00:21.315 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T15:00:21.317 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T15:00:21.317 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T15:00:21.317 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T15:00:21.317 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.317 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:21.318 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T15:00:21.319 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: jq-1.6-16.el9.x86_64 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.320 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T15:00:21.321 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T15:00:21.322 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T15:00:21.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T15:00:21.324 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T15:00:21.325 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T15:00:21.326 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T15:00:21.326 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T15:00:21.326 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T15:00:21.326 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T15:00:21.326 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T15:00:21.326 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T15:00:21.326 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T15:00:21.326 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.327 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T15:00:21.328 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T15:00:21.329 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.329 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.329 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:21.329 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T15:00:21.329 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T15:00:21.329 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T15:00:21.329 INFO:teuthology.orchestra.run.smithi186.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T15:00:21.329 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T15:00:21.329 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T15:00:21.330 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T15:00:21.330 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:21.330 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-28T15:00:21.851 DEBUG:teuthology.parallel:result is None 2024-03-28T15:00:22.775 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T15:00:22.776 INFO:teuthology.orchestra.run.smithi191.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T15:00:22.777 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.778 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.779 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-shell-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top-2:19.0.0-2499.ge0e48ba6.el9.noarch 2024-03-28T15:00:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T15:00:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T15:00:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T15:00:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T15:00:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T15:00:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T15:00:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T15:00:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T15:00:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: jq-1.6-16.el9.x86_64 2024-03-28T15:00:22.780 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T15:00:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T15:00:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T15:00:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T15:00:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T15:00:22.781 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T15:00:22.782 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T15:00:22.783 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T15:00:22.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T15:00:22.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T15:00:22.786 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.787 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T15:00:22.788 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T15:00:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:19.0.0-2499.ge0e48ba6.el9.x86_64 2024-03-28T15:00:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T15:00:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T15:00:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T15:00:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T15:00:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T15:00:22.789 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T15:00:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T15:00:22.790 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:22.790 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-28T15:00:23.305 DEBUG:teuthology.parallel:result is None 2024-03-28T15:00:23.306 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T15:00:23.536 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T15:00:23.563 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-2499.ge0e48ba6.el9 2024-03-28T15:00:23.563 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2499.ge0e48ba6.el9 2024-03-28T15:00:23.563 INFO:teuthology.task.install:The correct ceph version 19.0.0-2499.ge0e48ba6 is installed. 2024-03-28T15:00:23.565 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T15:00:23.735 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T15:00:23.759 INFO:teuthology.orchestra.run.smithi191.stdout:19.0.0-2499.ge0e48ba6.el9 2024-03-28T15:00:23.759 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2499.ge0e48ba6.el9 2024-03-28T15:00:23.760 INFO:teuthology.task.install:The correct ceph version 19.0.0-2499.ge0e48ba6 is installed. 2024-03-28T15:00:23.761 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T15:00:23.762 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:23.762 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T15:00:23.797 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:23.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T15:00:23.838 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T15:00:23.838 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:23.839 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T15:00:23.871 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T15:00:23.947 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:23.947 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T15:00:23.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T15:00:24.049 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T15:00:24.049 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:24.049 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T15:00:24.081 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T15:00:24.154 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:24.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T15:00:24.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T15:00:24.256 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T15:00:24.256 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:24.256 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T15:00:24.288 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T15:00:24.359 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:24.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T15:00:24.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T15:00:24.458 INFO:teuthology.run_tasks:Running task ceph... 2024-03-28T15:00:24.543 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-28T15:00:24.543 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /var/log/ceph 2024-03-28T15:00:24.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2024-03-28T15:00:24.575 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-28T15:00:24.575 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-28T15:00:24.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-28T15:00:24.644 INFO:tasks.ceph:Creating extra log directories... 2024-03-28T15:00:24.644 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-28T15:00:24.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-28T15:00:24.720 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-28T15:00:24.720 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-28T15:00:24.720 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-28T15:00:24.721 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:24.721 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-28T15:00:24.768 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-28T15:00:24.884 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-28T15:00:24.911 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:24.911 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-28T15:00:24.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-28T15:00:25.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-28T15:00:25.080 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-28T15:00:25.081 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-28T15:00:25.081 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984', 'branch': 'wip-batrick-testing-20240327.230800', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7627984', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240327.230800'}, 'ceph': {'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240327.230800', 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 7752, 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15', 'sleep_before_teardown': 0, 'subset': '1/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240327.230800', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi186.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0PFIEkLyGui6wXmHsGy8PaMcdiY15HyvBvZeH/tU5plzXHweD4TBjDdVh/wR+38K6IBCIsbrxOSnb2KhpgaI4=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPl/NaYLYsV2jUAwfErgcSHMcwl9tx+xvwOnB5XTNmoI3kMfLwxGPgh4IyYfUkr9ouQzNjpKWUKCBhM0MBiE2I='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-28_07:14:07', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-28T15:00:25.081 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-28T15:00:25.084 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-28T15:00:25.130 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/run/ceph 2024-03-28T15:00:25.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2024-03-28T15:00:25.197 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:25.197 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T15:00:25.220 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T15:00:25.220 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-03-28T15:00:25.275 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T15:00:25.275 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:00:25.275 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-28T15:00:25.275 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:00:25.275 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:00:25.275 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-28 15:00:13.420561287 +0000 2024-03-28T15:00:25.275 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-28 14:57:47.519902825 +0000 2024-03-28T15:00:25.275 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-28 14:57:47.519902825 +0000 2024-03-28T15:00:25.275 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-28 14:57:47.519902825 +0000 2024-03-28T15:00:25.276 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T15:00:25.340 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-28T15:00:25.341 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-28T15:00:25.341 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000198778 s, 2.6 MB/s 2024-03-28T15:00:25.341 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T15:00:25.397 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-03-28T15:00:25.452 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T15:00:25.453 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:00:25.453 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-28T15:00:25.453 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:00:25.453 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:00:25.453 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-28 15:00:13.420561287 +0000 2024-03-28T15:00:25.453 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-28 14:57:47.921895575 +0000 2024-03-28T15:00:25.453 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-28 14:57:47.921895575 +0000 2024-03-28T15:00:25.453 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-28 14:57:47.921895575 +0000 2024-03-28T15:00:25.454 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T15:00:25.518 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-28T15:00:25.519 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-28T15:00:25.519 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000184462 s, 2.8 MB/s 2024-03-28T15:00:25.520 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T15:00:25.575 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-03-28T15:00:25.630 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T15:00:25.630 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:00:25.631 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-28T15:00:25.631 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:00:25.631 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:00:25.631 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-28 15:00:13.421561269 +0000 2024-03-28T15:00:25.631 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-28 14:57:48.285889010 +0000 2024-03-28T15:00:25.631 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-28 14:57:48.285889010 +0000 2024-03-28T15:00:25.631 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-28 14:57:48.285889010 +0000 2024-03-28T15:00:25.631 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T15:00:25.695 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-28T15:00:25.696 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-28T15:00:25.696 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000215322 s, 2.4 MB/s 2024-03-28T15:00:25.697 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T15:00:25.752 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-03-28T15:00:25.807 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T15:00:25.808 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:00:25.808 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-28T15:00:25.808 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:00:25.808 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:00:25.808 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-28 15:00:13.421561269 +0000 2024-03-28T15:00:25.808 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-28 14:57:48.637882662 +0000 2024-03-28T15:00:25.808 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-28 14:57:48.637882662 +0000 2024-03-28T15:00:25.808 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-28 14:57:48.637882662 +0000 2024-03-28T15:00:25.808 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T15:00:25.873 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-28T15:00:25.873 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-28T15:00:25.874 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000185965 s, 2.8 MB/s 2024-03-28T15:00:25.875 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T15:00:25.930 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T15:00:25.930 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:25.930 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T15:00:25.948 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T15:00:25.949 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-03-28T15:00:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T15:00:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:00:26.007 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-28T15:00:26.007 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:00:26.007 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:00:26.007 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-28 15:00:14.031067149 +0000 2024-03-28T15:00:26.007 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-28 14:57:47.508522218 +0000 2024-03-28T15:00:26.007 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-28 14:57:47.508522218 +0000 2024-03-28T15:00:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-28 14:57:47.508522218 +0000 2024-03-28T15:00:26.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T15:00:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-28T15:00:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-28T15:00:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000242531 s, 2.1 MB/s 2024-03-28T15:00:26.079 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T15:00:26.138 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-03-28T15:00:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T15:00:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:00:26.196 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-28T15:00:26.196 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:00:26.196 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:00:26.196 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-28 15:00:14.031067149 +0000 2024-03-28T15:00:26.196 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-28 14:57:47.901514973 +0000 2024-03-28T15:00:26.196 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-28 14:57:47.901514973 +0000 2024-03-28T15:00:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-28 14:57:47.901514973 +0000 2024-03-28T15:00:26.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T15:00:26.263 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-28T15:00:26.264 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-28T15:00:26.264 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000219279 s, 2.3 MB/s 2024-03-28T15:00:26.265 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T15:00:26.323 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-03-28T15:00:26.378 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T15:00:26.379 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:00:26.379 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-28T15:00:26.379 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:00:26.379 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:00:26.379 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-28 15:00:14.032067131 +0000 2024-03-28T15:00:26.379 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-28 14:57:48.274508096 +0000 2024-03-28T15:00:26.379 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-28 14:57:48.274508096 +0000 2024-03-28T15:00:26.379 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-28 14:57:48.274508096 +0000 2024-03-28T15:00:26.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T15:00:26.445 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-28T15:00:26.445 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-28T15:00:26.445 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00022963 s, 2.2 MB/s 2024-03-28T15:00:26.446 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T15:00:26.503 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-03-28T15:00:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T15:00:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:00:26.559 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-28T15:00:26.559 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:00:26.559 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:00:26.559 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-28 15:00:14.032067131 +0000 2024-03-28T15:00:26.559 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-28 14:57:48.629501551 +0000 2024-03-28T15:00:26.559 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-28 14:57:48.629501551 +0000 2024-03-28T15:00:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-28 14:57:48.629501551 +0000 2024-03-28T15:00:26.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T15:00:26.624 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-28T15:00:26.624 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-28T15:00:26.624 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000167137 s, 3.1 MB/s 2024-03-28T15:00:26.625 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T15:00:26.681 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T15:00:26.682 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-28T15:00:26.682 INFO:tasks.ceph:Generating config... 2024-03-28T15:00:26.683 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-28T15:00:26.683 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-03-28T15:00:26.683 INFO:tasks.ceph:[client] debug client = 20 2024-03-28T15:00:26.683 INFO:tasks.ceph:[client] debug ms = 1 2024-03-28T15:00:26.683 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-28T15:00:26.683 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-28T15:00:26.683 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-28T15:00:26.683 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = False 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-28T15:00:26.684 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-28T15:00:26.685 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-28T15:00:26.685 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-28T15:00:26.685 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-28T15:00:26.685 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-28T15:00:26.685 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-28T15:00:26.685 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-03-28T15:00:26.685 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-28T15:00:26.685 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-28T15:00:26.685 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-28T15:00:26.685 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-28T15:00:26.685 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-28T15:00:26.685 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-28T15:00:26.685 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-28T15:00:26.685 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-28T15:00:26.686 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-28T15:00:26.686 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-28T15:00:26.686 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-28T15:00:26.686 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-28T15:00:26.686 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-28T15:00:26.686 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-28T15:00:26.686 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-28T15:00:26.686 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-28T15:00:26.686 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-28T15:00:26.686 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-28T15:00:26.686 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-28T15:00:26.686 INFO:tasks.ceph:Setting up mon.a... 2024-03-28T15:00:26.687 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-28T15:00:26.744 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.keyring 2024-03-28T15:00:26.747 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-28T15:00:26.794 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T15:00:26.821 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.186'), ('mon.b', '172.21.15.191'), ('mon.c', '[v2:172.21.15.191:3301,v1:172.21.15.191:6790]')] 2024-03-28T15:00:26.821 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.186,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-28T15:00:26.822 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:26.822 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T15:00:26.876 DEBUG:teuthology.orchestra.run.smithi186:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.186 --add b 172.21.15.191 --addv c '[v2:172.21.15.191:3301,v1:172.21.15.191:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:26.963 INFO:teuthology.orchestra.run.smithi186.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:26.963 INFO:teuthology.orchestra.run.smithi186.stdout:monmaptool: generated fsid 694dd083-ff98-42dc-bf10-681382259053 2024-03-28T15:00:26.963 INFO:teuthology.orchestra.run.smithi186.stdout:setting min_mon_release = quincy 2024-03-28T15:00:26.963 INFO:teuthology.orchestra.run.smithi186.stdout:epoch 0 2024-03-28T15:00:26.963 INFO:teuthology.orchestra.run.smithi186.stdout:fsid 694dd083-ff98-42dc-bf10-681382259053 2024-03-28T15:00:26.963 INFO:teuthology.orchestra.run.smithi186.stdout:last_changed 2024-03-28T15:00:26.964087+0000 2024-03-28T15:00:26.963 INFO:teuthology.orchestra.run.smithi186.stdout:created 2024-03-28T15:00:26.964087+0000 2024-03-28T15:00:26.963 INFO:teuthology.orchestra.run.smithi186.stdout:min_mon_release 17 (quincy) 2024-03-28T15:00:26.963 INFO:teuthology.orchestra.run.smithi186.stdout:election_strategy: 1 2024-03-28T15:00:26.963 INFO:teuthology.orchestra.run.smithi186.stdout:0: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.a 2024-03-28T15:00:26.964 INFO:teuthology.orchestra.run.smithi186.stdout:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-28T15:00:26.964 INFO:teuthology.orchestra.run.smithi186.stdout:2: [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] mon.c 2024-03-28T15:00:26.964 INFO:teuthology.orchestra.run.smithi186.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-28T15:00:26.965 DEBUG:teuthology.orchestra.run.smithi186:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T15:00:26.979 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 694dd083-ff98-42dc-bf10-681382259053... 2024-03-28T15:00:26.980 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-28T15:00:27.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-28T15:00:27.070 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2024-03-28T15:00:27.070 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.071 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2024-03-28T15:00:27.072 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = "172.21.15.186,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]" 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = 694dd083-ff98-42dc-bf10-681382259053 2024-03-28T15:00:27.073 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2024-03-28T15:00:27.074 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug trim objects = true 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2024-03-28T15:00:27.075 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: bdev async discard = True 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: bdev enable discard = True 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore allocator = bitmap 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 1/20 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 1/20 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 4/10 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2024-03-28T15:00:27.076 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: osd blocked scrub grace period = 3600 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: osd max markdown count = 1000 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-03-28T15:00:27.077 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: debug client = 20 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2024-03-28T15:00:27.078 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2024-03-28T15:00:27.079 INFO:teuthology.orchestra.run.smithi186.stdout: mon down mkfs grace = 300 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: mon op complaint time = 120 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: client mount timeout = 600 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: client_shutdown_timeout = 120 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: debug client = 20 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2024-03-28T15:00:27.080 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.b] 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.c] 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.e] 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.f] 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout:[mds] 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds = 20 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds balancer = 20 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal fragment size max = 10000 2024-03-28T15:00:27.081 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal merge size = 5 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal split bits = 3 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal split size = 100 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug frag = True 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug scatterstat = True 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: mds oft prefetch dirfrags = False 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: mds op complaint time = 180 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: mds verify scatter = True 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: mds_dir_prefetch = True 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: mds_sleep_rank_change = 5000000.0 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2024-03-28T15:00:27.082 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2024-03-28T15:00:27.083 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2024-03-28T15:00:27.083 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2024-03-28T15:00:27.083 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-28T15:00:27.083 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2024-03-28T15:00:27.083 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.083 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2024-03-28T15:00:27.083 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.083 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = "172.21.15.186,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]" 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = 694dd083-ff98-42dc-bf10-681382259053 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2024-03-28T15:00:27.086 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug trim objects = true 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.087 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: bdev async discard = True 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: bdev enable discard = True 2024-03-28T15:00:27.088 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore allocator = bitmap 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 1/20 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 1/20 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 4/10 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: osd blocked scrub grace period = 3600 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: osd max markdown count = 1000 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2024-03-28T15:00:27.089 INFO:teuthology.orchestra.run.smithi191.stdout: osd op complaint time = 180 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: debug client = 20 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-28T15:00:27.090 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.091 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: mon down mkfs grace = 300 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: mon op complaint time = 120 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2024-03-28T15:00:27.092 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout: client mount timeout = 600 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout: client_shutdown_timeout = 120 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout: debug client = 20 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout: rados mon op timeout = 900 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout: rados osd op timeout = 900 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.b] 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.c] 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.a] 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.c] 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.e] 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.b] 2024-03-28T15:00:27.093 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.d] 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.f] 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout:[mds] 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: debug mds = 20 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: debug mds balancer = 20 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal fragment size max = 10000 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal merge size = 5 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal split bits = 3 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal split size = 100 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: mds debug frag = True 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: mds debug scatterstat = True 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: mds oft prefetch dirfrags = False 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: mds op complaint time = 180 2024-03-28T15:00:27.094 INFO:teuthology.orchestra.run.smithi191.stdout: mds verify scatter = True 2024-03-28T15:00:27.095 INFO:teuthology.orchestra.run.smithi191.stdout: mds_dir_prefetch = True 2024-03-28T15:00:27.095 INFO:teuthology.orchestra.run.smithi191.stdout: mds_sleep_rank_change = 5000000.0 2024-03-28T15:00:27.095 INFO:teuthology.orchestra.run.smithi191.stdout: osd op complaint time = 180 2024-03-28T15:00:27.095 INFO:teuthology.orchestra.run.smithi191.stdout: rados mon op timeout = 900 2024-03-28T15:00:27.095 INFO:teuthology.orchestra.run.smithi191.stdout: rados osd op timeout = 900 2024-03-28T15:00:27.096 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-28T15:00:27.096 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-28T15:00:27.193 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-28T15:00:27.193 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:27.193 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-28T15:00:27.208 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:27.208 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-28T15:00:27.264 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T15:00:27.264 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:27.265 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-28T15:00:27.265 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T15:00:27.344 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:27.345 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:27.400 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2024-03-28T15:00:27.400 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:27.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-28T15:00:27.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T15:00:27.443 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:27.444 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:27.500 INFO:tasks.ceph:Setting up mon nodes... 2024-03-28T15:00:27.500 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-28T15:00:27.500 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-28T15:00:27.554 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-28T15:00:27.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T15:00:27.621 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T15:00:27.624 INFO:tasks.ceph:Setting up mds nodes... 2024-03-28T15:00:27.624 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-28T15:00:27.679 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-28T15:00:27.681 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-28T15:00:27.708 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-28T15:00:27.804 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-28T15:00:27.806 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-28T15:00:27.833 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-03-28T15:00:27.927 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-03-28T15:00:27.930 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-03-28T15:00:27.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-28T15:00:28.015 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-28T15:00:28.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-28T15:00:28.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-28T15:00:28.141 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-28T15:00:28.143 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-28T15:00:28.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-03-28T15:00:28.265 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-03-28T15:00:28.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-03-28T15:00:28.296 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-28T15:00:28.297 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T15:00:28.339 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-28T15:00:28.355 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-28T15:00:28.355 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-28T15:00:28.355 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-28T15:00:28.421 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T15:00:28.421 INFO:tasks.ceph:role: osd.0 2024-03-28T15:00:28.421 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T15:00:28.422 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-28T15:00:28.496 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T15:00:28.496 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T15:00:28.496 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T15:00:28.496 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T15:00:28.496 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T15:00:28.496 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-03-28T15:00:28.496 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T15:00:28.496 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T15:00:28.496 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T15:00:28.496 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T15:00:28.998 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-03-28T15:00:29.000 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-03-28T15:00:29.000 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-28T15:00:29.133 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-28T15:00:29.168 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-28T15:00:29.236 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T15:00:29.236 INFO:tasks.ceph:role: osd.1 2024-03-28T15:00:29.237 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T15:00:29.237 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-28T15:00:29.312 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T15:00:29.312 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T15:00:29.312 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T15:00:29.312 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T15:00:29.312 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T15:00:29.312 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-03-28T15:00:29.312 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T15:00:29.312 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T15:00:29.312 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T15:00:29.312 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T15:00:29.772 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-03-28T15:00:29.774 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-03-28T15:00:29.774 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-28T15:00:29.865 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-28T15:00:29.897 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-28T15:00:29.964 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T15:00:29.964 INFO:tasks.ceph:role: osd.2 2024-03-28T15:00:29.964 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T15:00:29.964 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-28T15:00:30.040 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T15:00:30.040 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T15:00:30.040 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T15:00:30.040 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T15:00:30.040 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T15:00:30.040 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-03-28T15:00:30.040 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T15:00:30.040 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T15:00:30.040 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T15:00:30.040 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T15:00:30.529 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-03-28T15:00:30.531 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-03-28T15:00:30.531 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-28T15:00:30.669 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-28T15:00:30.701 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-28T15:00:30.768 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T15:00:30.768 INFO:tasks.ceph:role: osd.3 2024-03-28T15:00:30.768 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T15:00:30.768 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-28T15:00:30.842 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T15:00:30.843 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T15:00:30.843 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T15:00:30.843 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T15:00:30.843 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T15:00:30.843 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-03-28T15:00:30.843 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T15:00:30.843 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T15:00:30.843 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T15:00:30.843 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T15:00:31.350 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-03-28T15:00:31.351 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-03-28T15:00:31.352 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-28T15:00:31.476 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-28T15:00:31.508 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:31.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:31.605+0000 7f0028899640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-28T15:00:31.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:31.606+0000 7f0028899640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-28T15:00:31.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:31.606+0000 7f0028899640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T15:00:31.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:31.606+0000 7f0028899640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T15:00:31.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:31.606+0000 7f0028899640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T15:00:31.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:31.606+0000 7f0028899640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-28T15:00:34.461 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-28T15:00:34.490 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:34.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:34.577+0000 7f4e0da1c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-28T15:00:34.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:34.577+0000 7f4e0da1c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-28T15:00:34.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:34.577+0000 7f4e0da1c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T15:00:34.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:34.577+0000 7f4e0da1c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T15:00:34.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:34.577+0000 7f4e0da1c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T15:00:34.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:34.578+0000 7f4e0da1c640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-28T15:00:37.431 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-28T15:00:37.461 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:37.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:37.550+0000 7fe2fd824640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-28T15:00:37.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:37.550+0000 7fe2fd824640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-28T15:00:37.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:37.550+0000 7fe2fd824640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T15:00:37.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:37.550+0000 7fe2fd824640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T15:00:37.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:37.550+0000 7fe2fd824640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T15:00:37.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:37.551+0000 7fe2fd824640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-28T15:00:40.402 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-28T15:00:40.431 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:40.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:40.518+0000 7ff21e813640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-28T15:00:40.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:40.518+0000 7ff21e813640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-28T15:00:40.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:40.518+0000 7ff21e813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T15:00:40.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:40.519+0000 7ff21e813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T15:00:40.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:40.519+0000 7ff21e813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T15:00:40.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:00:40.519+0000 7ff21e813640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-28T15:00:43.376 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-28T15:00:43.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-28T15:00:43.439 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T15:00:43.439 INFO:tasks.ceph:role: osd.4 2024-03-28T15:00:43.439 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-28T15:00:43.439 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-28T15:00:43.516 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T15:00:43.517 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T15:00:43.517 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T15:00:43.517 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T15:00:43.517 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T15:00:43.517 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-28T15:00:43.517 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T15:00:43.517 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T15:00:43.517 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T15:00:43.517 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T15:00:44.014 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-28T15:00:44.019 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-28T15:00:44.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-28T15:00:44.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-28T15:00:44.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-28T15:00:44.202 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T15:00:44.202 INFO:tasks.ceph:role: osd.5 2024-03-28T15:00:44.202 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-28T15:00:44.202 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-28T15:00:44.277 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T15:00:44.278 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T15:00:44.278 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T15:00:44.278 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T15:00:44.278 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T15:00:44.278 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-28T15:00:44.278 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T15:00:44.278 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T15:00:44.278 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T15:00:44.278 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T15:00:44.724 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-28T15:00:44.727 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-28T15:00:44.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-28T15:00:44.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-28T15:00:44.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-28T15:00:44.916 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T15:00:44.916 INFO:tasks.ceph:role: osd.6 2024-03-28T15:00:44.916 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-28T15:00:44.916 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-28T15:00:44.993 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T15:00:44.994 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T15:00:44.994 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T15:00:44.994 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T15:00:44.994 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T15:00:44.994 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-28T15:00:44.994 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T15:00:44.994 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T15:00:44.995 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T15:00:44.995 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T15:00:45.492 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-28T15:00:45.494 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-28T15:00:45.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-28T15:00:45.598 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-28T15:00:45.631 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-28T15:00:45.700 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T15:00:45.700 INFO:tasks.ceph:role: osd.7 2024-03-28T15:00:45.700 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-28T15:00:45.700 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-28T15:00:45.776 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T15:00:45.776 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T15:00:45.776 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T15:00:45.776 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T15:00:45.776 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T15:00:45.776 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-28T15:00:45.776 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T15:00:45.777 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T15:00:45.777 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T15:00:45.777 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T15:00:46.292 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-28T15:00:46.294 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-28T15:00:46.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-28T15:00:46.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-28T15:00:46.474 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:46.574 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:46.574+0000 7f963e015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-28T15:00:46.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:46.579+0000 7f963e015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-28T15:00:46.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:46.579+0000 7f963e015640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T15:00:46.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:46.579+0000 7f963e015640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T15:00:46.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:46.579+0000 7f963e015640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T15:00:46.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:46.580+0000 7f963e015640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-28T15:00:49.436 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-28T15:00:49.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:49.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:49.559+0000 7f25151a8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-28T15:00:49.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:49.559+0000 7f25151a8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-28T15:00:49.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:49.559+0000 7f25151a8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T15:00:49.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:49.559+0000 7f25151a8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T15:00:49.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:49.559+0000 7f25151a8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T15:00:49.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:49.560+0000 7f25151a8640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-28T15:00:52.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-28T15:00:52.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:52.530 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:52.530+0000 7f7df4620640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-28T15:00:52.530 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:52.530+0000 7f7df4620640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-28T15:00:52.530 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:52.530+0000 7f7df4620640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T15:00:52.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:52.530+0000 7f7df4620640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T15:00:52.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:52.530+0000 7f7df4620640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T15:00:52.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:52.531+0000 7f7df4620640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-28T15:00:55.083 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:00:55.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:00:55.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-28T15:00:55.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:00:55.494 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:55.494+0000 7f865019e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-28T15:00:55.494 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:55.494+0000 7f865019e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-28T15:00:55.494 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:55.494+0000 7f865019e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T15:00:55.494 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:55.494+0000 7f865019e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T15:00:55.494 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:55.494+0000 7f865019e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T15:00:55.494 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:00:55.494+0000 7f865019e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-28T15:00:58.347 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-28T15:00:58.379 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-28T15:00:58.379 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:58.379 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-28T15:00:58.412 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:58.412 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-28T15:00:58.482 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:58.482 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-28T15:00:58.551 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:58.551 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-03-28T15:00:58.619 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:58.619 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-28T15:00:58.686 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:58.686 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-28T15:00:58.753 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:58.753 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-28T15:00:58.820 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:58.820 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-28T15:00:58.887 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:58.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-28T15:00:58.919 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:58.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-28T15:00:58.990 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:58.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-28T15:00:59.059 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:59.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-03-28T15:00:59.128 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:59.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-28T15:00:59.196 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:59.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-28T15:00:59.265 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:59.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-28T15:00:59.334 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:00:59.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-28T15:00:59.401 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:00:59.401 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-28T15:00:59.416 INFO:tasks.ceph:Adding keys to all mons... 2024-03-28T15:00:59.417 DEBUG:teuthology.orchestra.run.smithi186:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-28T15:00:59.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-28T15:00:59.482 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.y] 2024-03-28T15:00:59.483 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQALhgVmQG0hIRAA+7pPQ2tLktNNl1whf2fNyw== 2024-03-28T15:00:59.483 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2024-03-28T15:00:59.483 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQALhgVmzw6LKBAA/swmEkF0MHsFZaAz+f4oKw== 2024-03-28T15:00:59.483 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2024-03-28T15:00:59.483 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQALhgVm38z8LxAAWK2EiVou36pELAx9spKoGg== 2024-03-28T15:00:59.483 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.e] 2024-03-28T15:00:59.483 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQALhgVmJJNcNxAAcPqdYNbyYZ5Sxpsu3vZFbw== 2024-03-28T15:00:59.483 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.0] 2024-03-28T15:00:59.483 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAPhgVmPIUoJBAAMA3ecTDsjmK0yFF0Rj+kEQ== 2024-03-28T15:00:59.483 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.1] 2024-03-28T15:00:59.484 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAShgVmVZh4IhAAppSQelJoYhTZdjBfeGtvyw== 2024-03-28T15:00:59.484 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.2] 2024-03-28T15:00:59.484 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAVhgVmCBbaIBAAp96/41Qe1POI0GSaFmZR4w== 2024-03-28T15:00:59.484 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.3] 2024-03-28T15:00:59.484 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAYhgVmC0P5HhAAv/hM0trWsEm9S7tjHW31ew== 2024-03-28T15:00:59.484 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.x] 2024-03-28T15:00:59.484 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQALhgVmwuEdJRAALX0ErY9UtZiniRtpa5eeNg== 2024-03-28T15:00:59.484 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2024-03-28T15:00:59.484 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAMhgVmWDz+ABAA5TWOMbSbu4UFv6y3CerhHQ== 2024-03-28T15:00:59.485 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2024-03-28T15:00:59.485 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAMhgVmOzV6CBAAL83DhGv3c4QnSEnrvP2kpw== 2024-03-28T15:00:59.485 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.f] 2024-03-28T15:00:59.485 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAMhgVmw2XgDxAAzWph2SyYFLXFedX7LjB6pw== 2024-03-28T15:00:59.485 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.4] 2024-03-28T15:00:59.485 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAehgVm1zlCIhAAgAzbTL4yVE4aGO1FIWJAPA== 2024-03-28T15:00:59.485 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.5] 2024-03-28T15:00:59.485 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAhhgVmiXpfIRAAJ1U/T0h50Rt9N7okrQENDw== 2024-03-28T15:00:59.485 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.6] 2024-03-28T15:00:59.485 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAkhgVmA9uqHxAA8hybW3jGQyckyG0HNdDEZA== 2024-03-28T15:00:59.485 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.7] 2024-03-28T15:00:59.486 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAnhgVm6wx9HRAA5lvV0V6don4qEp8cm2uItA== 2024-03-28T15:00:59.486 INFO:teuthology.orchestra.run.smithi186.stdout:[client.0] 2024-03-28T15:00:59.486 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAMhgVmCHVUFBAA7M5sKtsDObQku0fSG0CdFg== 2024-03-28T15:00:59.488 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.y] 2024-03-28T15:00:59.488 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQALhgVmQG0hIRAA+7pPQ2tLktNNl1whf2fNyw== 2024-03-28T15:00:59.488 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.a] 2024-03-28T15:00:59.488 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQALhgVmzw6LKBAA/swmEkF0MHsFZaAz+f4oKw== 2024-03-28T15:00:59.488 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.c] 2024-03-28T15:00:59.488 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQALhgVm38z8LxAAWK2EiVou36pELAx9spKoGg== 2024-03-28T15:00:59.488 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.e] 2024-03-28T15:00:59.488 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQALhgVmJJNcNxAAcPqdYNbyYZ5Sxpsu3vZFbw== 2024-03-28T15:00:59.488 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2024-03-28T15:00:59.488 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAPhgVmPIUoJBAAMA3ecTDsjmK0yFF0Rj+kEQ== 2024-03-28T15:00:59.489 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2024-03-28T15:00:59.489 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAShgVmVZh4IhAAppSQelJoYhTZdjBfeGtvyw== 2024-03-28T15:00:59.489 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2024-03-28T15:00:59.489 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAVhgVmCBbaIBAAp96/41Qe1POI0GSaFmZR4w== 2024-03-28T15:00:59.489 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.3] 2024-03-28T15:00:59.489 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAYhgVmC0P5HhAAv/hM0trWsEm9S7tjHW31ew== 2024-03-28T15:00:59.489 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2024-03-28T15:00:59.489 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQALhgVmwuEdJRAALX0ErY9UtZiniRtpa5eeNg== 2024-03-28T15:00:59.489 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.b] 2024-03-28T15:00:59.489 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAMhgVmWDz+ABAA5TWOMbSbu4UFv6y3CerhHQ== 2024-03-28T15:00:59.490 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.d] 2024-03-28T15:00:59.490 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAMhgVmOzV6CBAAL83DhGv3c4QnSEnrvP2kpw== 2024-03-28T15:00:59.490 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.f] 2024-03-28T15:00:59.490 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAMhgVmw2XgDxAAzWph2SyYFLXFedX7LjB6pw== 2024-03-28T15:00:59.490 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.4] 2024-03-28T15:00:59.490 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAehgVm1zlCIhAAgAzbTL4yVE4aGO1FIWJAPA== 2024-03-28T15:00:59.490 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.5] 2024-03-28T15:00:59.490 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAhhgVmiXpfIRAAJ1U/T0h50Rt9N7okrQENDw== 2024-03-28T15:00:59.490 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.6] 2024-03-28T15:00:59.491 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAkhgVmA9uqHxAA8hybW3jGQyckyG0HNdDEZA== 2024-03-28T15:00:59.491 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.7] 2024-03-28T15:00:59.491 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAnhgVm6wx9HRAA5lvV0V6don4qEp8cm2uItA== 2024-03-28T15:00:59.491 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2024-03-28T15:00:59.491 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAMhgVmCHVUFBAA7M5sKtsDObQku0fSG0CdFg== 2024-03-28T15:00:59.491 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T15:00:59.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T15:00:59.584 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:00:59.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:00:59.642 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:00:59.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:00:59.700 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:00:59.703 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:00:59.758 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:00:59.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:00:59.812 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:00:59.814 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:00:59.865 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:00:59.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:00:59.921 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:00:59.923 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:00:59.976 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T15:00:59.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T15:01:00.031 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:01:00.033 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:01:00.087 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:01:00.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:01:00.144 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:01:00.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T15:01:00.201 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:01:00.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:01:00.254 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:01:00.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:01:00.309 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:01:00.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:01:00.363 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:01:00.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T15:01:00.422 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T15:01:00.431 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T15:01:00.495 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-28T15:01:00.495 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-28T15:01:00.529 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T15:01:00.635 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-28T15:01:00.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-28T15:01:00.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T15:01:00.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-28T15:01:00.821 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-28T15:01:00.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T15:01:00.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-28T15:01:01.020 DEBUG:teuthology.orchestra.run.smithi186:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:01:01.023 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-28T15:01:01.076 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-28T15:01:01.077 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-28T15:01:01.077 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-28T15:01:01.080 INFO:tasks.ceph.mon.a:Started 2024-03-28T15:01:01.081 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-28T15:01:01.081 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-28T15:01:01.122 INFO:tasks.ceph.mon.b:Started 2024-03-28T15:01:01.122 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-28T15:01:01.122 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-28T15:01:01.126 INFO:tasks.ceph.mon.c:Started 2024-03-28T15:01:01.126 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-28T15:01:01.127 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-28T15:01:01.127 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-28T15:01:01.130 INFO:tasks.ceph.mgr.y:Started 2024-03-28T15:01:01.130 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-28T15:01:01.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-28T15:01:01.133 INFO:tasks.ceph.mgr.x:Started 2024-03-28T15:01:01.134 INFO:tasks.ceph:Setting crush tunables to default 2024-03-28T15:01:01.134 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd crush tunables default 2024-03-28T15:01:01.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.235+0000 7fb3756fa640 1 Processor -- start 2024-03-28T15:01:01.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.235+0000 7fb3756fa640 1 -- start start 2024-03-28T15:01:01.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.235+0000 7fb3756fa640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:01.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.235+0000 7fb3756fa640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb37015de90 con 0x7fb3700cce90 2024-03-28T15:01:01.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.236+0000 7fb3756fa640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3700cb6d0 con 0x7fb37015b2c0 2024-03-28T15:01:01.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.236+0000 7fb3756fa640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb3700c5ae0 con 0x7fb370154a10 2024-03-28T15:01:01.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.236+0000 7fb36effd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:01.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.236+0000 7fb36f7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fb3700cce90 0x7fb370165180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49344/0 (socket says 172.21.15.186:49344) 2024-03-28T15:01:01.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.236+0000 7fb36f7fe640 1 -- 172.21.15.186:0/3923237488 learned_addr learned my addr 172.21.15.186:0/3923237488 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:01.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.236+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 msgr2=0x7fb3700cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T15:01:01.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.236+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 msgr2=0x7fb3700cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:01.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.236+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:01.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.236+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:01:01.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.437+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:01.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.437+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 msgr2=0x7fb3700cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T15:01:01.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.437+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 msgr2=0x7fb3700cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:01.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.437+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:01.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.437+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:01:01.838 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.838+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:01.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.838+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 msgr2=0x7fb3700cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T15:01:01.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.838+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 msgr2=0x7fb3700cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:01.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.838+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:01.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:01.838+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T15:01:02.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:02.639+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:02.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:02.639+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 msgr2=0x7fb3700cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T15:01:02.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:02.639+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 msgr2=0x7fb3700cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:02.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:02.639+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:02.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:02.639+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36cff9640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 msgr2=0x7fb3700cb0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36cff9640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.186:3300/0 conn(0x7fb370154a10 0x7fb3700cb0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36cff9640 1 -- 172.21.15.186:0/3923237488 >> v1:172.21.15.191:6789/0 conn(0x7fb37015b2c0 legacy=0x7fb3700c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36cff9640 1 -- 172.21.15.186:0/3923237488 >> v1:172.21.15.186:6789/0 conn(0x7fb3700cce90 legacy=0x7fb370165180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36cff9640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36cff9640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36cff9640 1 -- 172.21.15.186:0/3923237488 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fb3700c5ae0 con 0x7fb35c006280 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36cff9640 1 -- 172.21.15.186:0/3923237488 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb35c008df0 con 0x7fb35c003710 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36e7fc640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 msgr2=0x7fb35c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T15:01:04.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36e7fc640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 msgr2=0x7fb35c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:04.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:04.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 msgr2=0x7fb35c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T15:01:04.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 msgr2=0x7fb35c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:04.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:04.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:01:04.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.236+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:01:04.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.437+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:04.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.437+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:04.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.437+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 msgr2=0x7fb35c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T15:01:04.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.437+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 msgr2=0x7fb35c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:04.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.437+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:04.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.437+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:01:04.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.437+0000 7fb36e7fc640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 msgr2=0x7fb35c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T15:01:04.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.437+0000 7fb36e7fc640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 msgr2=0x7fb35c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:04.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.437+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:04.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.437+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:01:04.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.838+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:04.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.838+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:04.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.838+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 msgr2=0x7fb35c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T15:01:04.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.838+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 msgr2=0x7fb35c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:04.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.838+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:04.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.838+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T15:01:04.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.838+0000 7fb36e7fc640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 msgr2=0x7fb35c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T15:01:04.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.838+0000 7fb36e7fc640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 msgr2=0x7fb35c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:04.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.838+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:04.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:04.838+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T15:01:05.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:05.639+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:05.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:05.639+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:05.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:05.640+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 msgr2=0x7fb35c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T15:01:05.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:05.640+0000 7fb36effd640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 msgr2=0x7fb35c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:05.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:05.640+0000 7fb36e7fc640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 msgr2=0x7fb35c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T15:01:05.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:05.640+0000 7fb36e7fc640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 msgr2=0x7fb35c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T15:01:05.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:05.640+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:05.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:05.640+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:01:05.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:05.640+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T15:01:05.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:05.640+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T15:01:06.912 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:06.912+0000 7fed68a42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:01:06.967 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:06.966+0000 7fed68a42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:01:07.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36cff9640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 msgr2=0x7fb35c005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T15:01:07.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36cff9640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003710 0x7fb35c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36cff9640 1 -- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 msgr2=0x7fb35c008740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T15:01:07.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36cff9640 1 --2- 172.21.15.186:0/3923237488 >> v2:172.21.15.191:3300/0 conn(0x7fb35c006280 0x7fb35c008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36f7fe640 1 -- 172.21.15.186:0/3923237488 reap_dead start 2024-03-28T15:01:07.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36cff9640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003b50 0x7fb35c0066b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:07.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36cff9640 1 -- 172.21.15.186:0/3923237488 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb37015e100 con 0x7fb35c006bf0 2024-03-28T15:01:07.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36cff9640 1 -- 172.21.15.186:0/3923237488 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb35c00fc20 con 0x7fb35c008750 2024-03-28T15:01:07.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36cff9640 1 -- 172.21.15.186:0/3923237488 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb3700c5ae0 con 0x7fb35c003b50 2024-03-28T15:01:07.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3695785763 0 0) 0x7fb37015e100 con 0x7fb35c006bf0 2024-03-28T15:01:07.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb358003520 con 0x7fb35c006bf0 2024-03-28T15:01:07.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003b50 0x7fb35c0066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:07.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3894912284 0 0) 0x7fb358003520 con 0x7fb35c006bf0 2024-03-28T15:01:07.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003b50 msgr2=0x7fb35c0066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:07.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36dffb640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003b50 0x7fb35c0066b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 >> v1:172.21.15.191:6789/0 conn(0x7fb35c008750 legacy=0x7fb35c00d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c010ee0 con 0x7fb35c006bf0 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7fb364007da0 con 0x7fb35c006bf0 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.236+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 >> v1:172.21.15.186:6789/0 conn(0x7fb35c006bf0 legacy=0x7fb35c007030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36dffb640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3580039e0 0x7fb358003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36dffb640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb358004800 0x7fb358004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36dffb640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003b50 0x7fb358005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb3700c5ae0 con 0x7fb358004800 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb35800bb50 con 0x7fb3580039e0 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb35800bd50 con 0x7fb35c003b50 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb358004800 0x7fb358004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3580039e0 0x7fb358003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36f7fe640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003b50 0x7fb358005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36e7fc640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003b50 msgr2=0x7fb358005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:07.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb35c003b50 0x7fb358005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36e7fc640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3580039e0 msgr2=0x7fb358003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3580039e0 0x7fb358003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36e7fc640 1 -- 172.21.15.186:0/3923237488 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb358003520 con 0x7fb358004800 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.237+0000 7fb36effd640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3580039e0 0x7fb358003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.238+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb358004800 0x7fb358004c80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb3640039f0 tx=0x7fb358005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46b4adbee54ed70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.238+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364032070 con 0x7fb358004800 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.238+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb364007230 con 0x7fb358004800 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.238+0000 7fb36dffb640 1 -- 172.21.15.186:0/3923237488 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3640036d0 con 0x7fb358004800 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.238+0000 7fb3756fa640 1 -- 172.21.15.186:0/3923237488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb358004800 msgr2=0x7fb358004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.238+0000 7fb3756fa640 1 --2- 172.21.15.186:0/3923237488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb358004800 0x7fb358004c80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb3640039f0 tx=0x7fb358005830 comp rx=0 tx=0).stop 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.238+0000 7fb36f7fe640 1 -- 172.21.15.186:0/3923237488 reap_dead start 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.239+0000 7fb3756fa640 1 -- 172.21.15.186:0/3923237488 shutdown_connections 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.239+0000 7fb3756fa640 1 -- 172.21.15.186:0/3923237488 >> 172.21.15.186:0/3923237488 conn(0x7fb37006b640 msgr2=0x7fb37006ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:07.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.239+0000 7fb3756fa640 1 -- 172.21.15.186:0/3923237488 shutdown_connections 2024-03-28T15:01:07.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.239+0000 7fb3756fa640 1 -- 172.21.15.186:0/3923237488 wait complete. 2024-03-28T15:01:07.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.240+0000 7fb3756fa640 1 Processor -- start 2024-03-28T15:01:07.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.240+0000 7fb3756fa640 1 -- start start 2024-03-28T15:01:07.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb3756fa640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb358004800 0x7fb37015cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:07.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb3756fa640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3700b5850 0x7fb3701f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:07.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb3756fa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37014e4b0 0x7fb3701f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:07.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb3756fa640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb3700cb9b0 con 0x7fb37014e4b0 2024-03-28T15:01:07.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb3756fa640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb37015e750 con 0x7fb358004800 2024-03-28T15:01:07.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb3756fa640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb37015b710 con 0x7fb3700b5850 2024-03-28T15:01:07.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb36f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37014e4b0 0x7fb3701f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb36f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37014e4b0 0x7fb3701f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:35262/0 (socket says 172.21.15.186:35262) 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb36f7fe640 1 -- 172.21.15.186:0/216056062 learned_addr learned my addr 172.21.15.186:0/216056062 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb36effd640 1 --2- 172.21.15.186:0/216056062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb358004800 0x7fb37015cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb36e7fc640 1 --2- 172.21.15.186:0/216056062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3700b5850 0x7fb3701f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb36f7fe640 1 -- 172.21.15.186:0/216056062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3700b5850 msgr2=0x7fb3701f3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb36f7fe640 1 --2- 172.21.15.186:0/216056062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3700b5850 0x7fb3701f3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb36f7fe640 1 -- 172.21.15.186:0/216056062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb358004800 msgr2=0x7fb37015cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb36f7fe640 1 --2- 172.21.15.186:0/216056062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb358004800 0x7fb37015cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb36f7fe640 1 -- 172.21.15.186:0/216056062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb370150ae0 con 0x7fb37014e4b0 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb36f7fe640 1 --2- 172.21.15.186:0/216056062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37014e4b0 0x7fb3701f43b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb35c008d00 tx=0x7fb368007a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb3577fe640 1 -- 172.21.15.186:0/216056062 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb368004b60 con 0x7fb37014e4b0 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb3577fe640 1 -- 172.21.15.186:0/216056062 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb368004d20 con 0x7fb37014e4b0 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb3756fa640 1 -- 172.21.15.186:0/216056062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3700b43e0 con 0x7fb37014e4b0 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb3577fe640 1 -- 172.21.15.186:0/216056062 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36800e8a0 con 0x7fb37014e4b0 2024-03-28T15:01:07.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.241+0000 7fb3756fa640 1 -- 172.21.15.186:0/216056062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3700b49a0 con 0x7fb37014e4b0 2024-03-28T15:01:07.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.242+0000 7fb3577fe640 1 -- 172.21.15.186:0/216056062 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb36801a030 con 0x7fb37014e4b0 2024-03-28T15:01:07.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.242+0000 7fb3577fe640 1 -- 172.21.15.186:0/216056062 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb36801c410 con 0x7fb37014e4b0 2024-03-28T15:01:07.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.243+0000 7fb3756fa640 1 -- 172.21.15.186:0/216056062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb370069570 con 0x7fb37014e4b0 2024-03-28T15:01:07.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.245+0000 7fb3577fe640 1 -- 172.21.15.186:0/216056062 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb368021020 con 0x7fb37014e4b0 2024-03-28T15:01:07.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.310+0000 7fb3756fa640 1 -- 172.21.15.186:0/216056062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fb3701532c0 con 0x7fb37014e4b0 2024-03-28T15:01:07.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.312+0000 7fb3577fe640 1 -- 172.21.15.186:0/216056062 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fb368013420 con 0x7fb37014e4b0 2024-03-28T15:01:07.312 INFO:teuthology.orchestra.run.smithi186.stderr:adjusted tunables profile to default 2024-03-28T15:01:07.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.313+0000 7fb3756fa640 1 -- 172.21.15.186:0/216056062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37014e4b0 msgr2=0x7fb3701f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:07.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.313+0000 7fb3756fa640 1 --2- 172.21.15.186:0/216056062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37014e4b0 0x7fb3701f43b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb35c008d00 tx=0x7fb368007a80 comp rx=0 tx=0).stop 2024-03-28T15:01:07.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.313+0000 7fb3756fa640 1 -- 172.21.15.186:0/216056062 shutdown_connections 2024-03-28T15:01:07.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.313+0000 7fb3756fa640 1 --2- 172.21.15.186:0/216056062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37014e4b0 0x7fb3701f43b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.313+0000 7fb3756fa640 1 --2- 172.21.15.186:0/216056062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3700b5850 0x7fb3701f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.313+0000 7fb3756fa640 1 --2- 172.21.15.186:0/216056062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb358004800 0x7fb37015cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.313+0000 7fb3756fa640 1 -- 172.21.15.186:0/216056062 >> 172.21.15.186:0/216056062 conn(0x7fb37006b640 msgr2=0x7fb3701537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:07.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.313+0000 7fb3756fa640 1 -- 172.21.15.186:0/216056062 shutdown_connections 2024-03-28T15:01:07.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:07.313+0000 7fb3756fa640 1 -- 172.21.15.186:0/216056062 wait complete. 2024-03-28T15:01:07.324 INFO:tasks.ceph:check_enable_crimson: False 2024-03-28T15:01:07.324 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-28T15:01:07.324 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:01:07.324 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-28T15:01:07.392 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:01:07.392 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-28T15:01:07.460 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:01:07.460 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-28T15:01:07.512 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:07.512+0000 7fed68a42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:01:07.527 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:01:07.527 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-28T15:01:07.569 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:07.568+0000 7fed68a42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:01:07.594 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:01:07.594 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-28T15:01:07.616 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:07.616+0000 7fed68a42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:01:07.621 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:01:07.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-28T15:01:07.671 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:07.671+0000 7fed68a42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:01:07.689 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:01:07.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-28T15:01:07.689 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:07.689+0000 7fb636a3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:01:07.714 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:07.714+0000 7fed68a42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:01:07.745 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:07.745+0000 7fb636a3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:01:07.755 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-28T15:01:07.755 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-28T15:01:07.822 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 1c924c36-8b44-4912-b678-273fc481ddc0 0 2024-03-28T15:01:07.829 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:07.829+0000 7fed68a42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:01:07.900 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:07.899+0000 7fed68a42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:01:07.946 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:07.945+0000 7fed68a42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:01:07.963 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.963+0000 7f2c817b6640 1 Processor -- start 2024-03-28T15:01:07.964 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.963+0000 7f2c817b6640 1 -- start start 2024-03-28T15:01:07.965 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.964+0000 7f2c817b6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2c7c06db50 0x7f2c7c06df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:07.965 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.964+0000 7f2c817b6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c7c0c8600 con 0x7f2c7c0c8110 2024-03-28T15:01:07.965 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.964+0000 7f2c817b6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c7c157740 con 0x7f2c7c151870 2024-03-28T15:01:07.965 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.964+0000 7f2c817b6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f2c7c158070 con 0x7f2c7c06db50 2024-03-28T15:01:07.965 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.964+0000 7f2c7b7fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2c7c06db50 0x7f2c7c06df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:07.966 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.964+0000 7f2c7b7fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2c7c06db50 0x7f2c7c06df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40638/0 (socket says 172.21.15.191:40638) 2024-03-28T15:01:07.966 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.964+0000 7f2c7b7fe640 1 -- 172.21.15.191:0/1642076656 learned_addr learned my addr 172.21.15.191:0/1642076656 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:07.966 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.965+0000 7f2c7b7fe640 1 -- 172.21.15.191:0/1642076656 >> v1:172.21.15.191:6789/0 conn(0x7f2c7c151870 legacy=0x7f2c7c0c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:07.966 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.965+0000 7f2c7b7fe640 1 -- 172.21.15.191:0/1642076656 >> v1:172.21.15.186:6789/0 conn(0x7f2c7c0c8110 legacy=0x7f2c7c06ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:07.966 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.965+0000 7f2c7b7fe640 1 -- 172.21.15.191:0/1642076656 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7c157a20 con 0x7f2c7c06db50 2024-03-28T15:01:07.966 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.965+0000 7f2c7b7fe640 1 --2- 172.21.15.191:0/1642076656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c7c06db50 0x7f2c7c06df90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009e50 tx=0x7f2c7402c090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7262d4f1a3be29a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:07.966 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.965+0000 7f2c79ffb640 1 -- 172.21.15.191:0/1642076656 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7402cc30 con 0x7f2c7c06db50 2024-03-28T15:01:07.966 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.965+0000 7f2c79ffb640 1 -- 172.21.15.191:0/1642076656 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c7402cdd0 con 0x7f2c7c06db50 2024-03-28T15:01:07.966 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.966+0000 7f2c79ffb640 1 -- 172.21.15.191:0/1642076656 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c74034610 con 0x7f2c7c06db50 2024-03-28T15:01:07.970 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.966+0000 7f2c817b6640 1 -- 172.21.15.191:0/1642076656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c7c06db50 msgr2=0x7f2c7c06df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:07.970 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.966+0000 7f2c817b6640 1 --2- 172.21.15.191:0/1642076656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c7c06db50 0x7f2c7c06df90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009e50 tx=0x7f2c7402c090 comp rx=0 tx=0).stop 2024-03-28T15:01:07.970 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.966+0000 7f2c817b6640 1 -- 172.21.15.191:0/1642076656 shutdown_connections 2024-03-28T15:01:07.970 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.966+0000 7f2c817b6640 1 --2- 172.21.15.191:0/1642076656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c7c06db50 0x7f2c7c06df90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.970 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.966+0000 7f2c817b6640 1 -- 172.21.15.191:0/1642076656 >> 172.21.15.191:0/1642076656 conn(0x7f2c7c06c010 msgr2=0x7f2c7c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:07.970 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.966+0000 7f2c817b6640 1 -- 172.21.15.191:0/1642076656 shutdown_connections 2024-03-28T15:01:07.970 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.966+0000 7f2c817b6640 1 -- 172.21.15.191:0/1642076656 wait complete. 2024-03-28T15:01:07.970 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.967+0000 7f2c817b6640 1 Processor -- start 2024-03-28T15:01:07.970 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.967+0000 7f2c817b6640 1 -- start start 2024-03-28T15:01:07.970 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c817b6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c7c06db50 0x7f2c7c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c817b6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c7c0c8110 0x7f2c7c154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c817b6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c7c151870 0x7f2c7c1f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c817b6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7c06f2f0 con 0x7f2c7c151870 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c817b6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7c151c50 con 0x7f2c7c0c8110 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c817b6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f2c7c0c4250 con 0x7f2c7c06db50 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c7affd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c7c06db50 0x7f2c7c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c7a7fc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c7c0c8110 0x7f2c7c154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c7affd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c7c06db50 0x7f2c7c06d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:33320/0 (socket says 172.21.15.191:33320) 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c7affd640 1 -- 172.21.15.191:0/3133193202 learned_addr learned my addr 172.21.15.191:0/3133193202 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c7b7fe640 1 --2- 172.21.15.191:0/3133193202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c7c151870 0x7f2c7c1f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c7a7fc640 1 -- 172.21.15.191:0/3133193202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c7c06db50 msgr2=0x7f2c7c06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c7a7fc640 1 --2- 172.21.15.191:0/3133193202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c7c06db50 0x7f2c7c06d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.972 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c7a7fc640 1 -- 172.21.15.191:0/3133193202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c7c151870 msgr2=0x7f2c7c1f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:07.972 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c7a7fc640 1 --2- 172.21.15.191:0/3133193202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c7c151870 0x7f2c7c1f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:07.972 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c7a7fc640 1 -- 172.21.15.191:0/3133193202 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7c1f1b20 con 0x7f2c7c0c8110 2024-03-28T15:01:07.972 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.968+0000 7f2c7affd640 1 --2- 172.21.15.191:0/3133193202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c7c06db50 0x7f2c7c06d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:07.972 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.969+0000 7f2c7a7fc640 1 --2- 172.21.15.191:0/3133193202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c7c0c8110 0x7f2c7c154320 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00c310 tx=0x7f2c6c00c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:07.972 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.969+0000 7f2c637fe640 1 -- 172.21.15.191:0/3133193202 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c0052d0 con 0x7f2c7c0c8110 2024-03-28T15:01:07.972 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.969+0000 7f2c637fe640 1 -- 172.21.15.191:0/3133193202 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c6c005490 con 0x7f2c7c0c8110 2024-03-28T15:01:07.972 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.969+0000 7f2c637fe640 1 -- 172.21.15.191:0/3133193202 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c007260 con 0x7f2c7c0c8110 2024-03-28T15:01:07.972 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.969+0000 7f2c817b6640 1 -- 172.21.15.191:0/3133193202 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c7c06d860 con 0x7f2c7c0c8110 2024-03-28T15:01:07.972 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.969+0000 7f2c817b6640 1 -- 172.21.15.191:0/3133193202 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c7c1eea80 con 0x7f2c7c0c8110 2024-03-28T15:01:07.972 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.969+0000 7f2c637fe640 1 -- 172.21.15.191:0/3133193202 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2c6c007400 con 0x7f2c7c0c8110 2024-03-28T15:01:07.973 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.969+0000 7f2c637fe640 1 -- 172.21.15.191:0/3133193202 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f2c6c003b20 con 0x7f2c7c0c8110 2024-03-28T15:01:07.973 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.970+0000 7f2c817b6640 1 -- 172.21.15.191:0/3133193202 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c7c073e10 con 0x7f2c7c0c8110 2024-03-28T15:01:07.973 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:07.972+0000 7f2c637fe640 1 -- 172.21.15.191:0/3133193202 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2c7c073e10 con 0x7f2c7c0c8110 2024-03-28T15:01:07.998 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:07.998+0000 7fed68a42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:01:08.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.044+0000 7f2c817b6640 1 -- 172.21.15.191:0/3133193202 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1c924c36-8b44-4912-b678-273fc481ddc0", "id": 0} v 0) -- 0x7f2c7c069570 con 0x7f2c7c0c8110 2024-03-28T15:01:08.044 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:08.045+0000 7fed68a42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:01:08.047 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.047+0000 7f2c637fe640 1 -- 172.21.15.191:0/3133193202 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1c924c36-8b44-4912-b678-273fc481ddc0", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f2c6c003d20 con 0x7f2c7c0c8110 2024-03-28T15:01:08.047 INFO:teuthology.orchestra.run.smithi191.stdout:0 2024-03-28T15:01:08.048 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.048+0000 7f2c817b6640 1 -- 172.21.15.191:0/3133193202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c7c0c8110 msgr2=0x7f2c7c154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.048 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.048+0000 7f2c817b6640 1 --2- 172.21.15.191:0/3133193202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c7c0c8110 0x7f2c7c154320 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00c310 tx=0x7f2c6c00c7d0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.048 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.048+0000 7f2c817b6640 1 -- 172.21.15.191:0/3133193202 shutdown_connections 2024-03-28T15:01:08.048 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.048+0000 7f2c817b6640 1 --2- 172.21.15.191:0/3133193202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c7c151870 0x7f2c7c1f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.049 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.048+0000 7f2c817b6640 1 --2- 172.21.15.191:0/3133193202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c7c0c8110 0x7f2c7c154320 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.049 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.048+0000 7f2c817b6640 1 --2- 172.21.15.191:0/3133193202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c7c06db50 0x7f2c7c06d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.049 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.048+0000 7f2c817b6640 1 -- 172.21.15.191:0/3133193202 >> 172.21.15.191:0/3133193202 conn(0x7f2c7c06c010 msgr2=0x7f2c7c147b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:08.049 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.048+0000 7f2c817b6640 1 -- 172.21.15.191:0/3133193202 shutdown_connections 2024-03-28T15:01:08.049 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.048+0000 7f2c817b6640 1 -- 172.21.15.191:0/3133193202 wait complete. 2024-03-28T15:01:08.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 67303d31-dc97-42d2-98cf-a787593dadff 1 2024-03-28T15:01:08.160 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.160+0000 7f0d27264640 1 Processor -- start 2024-03-28T15:01:08.160 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.160+0000 7f0d27264640 1 -- start start 2024-03-28T15:01:08.163 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d27264640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0d201516e0 0x7f0d2006c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.163 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d27264640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0d20073a40 0x7f0d20157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.163 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d27264640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d200c8570 con 0x7f0d200c79a0 2024-03-28T15:01:08.163 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d27264640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f0d200c8320 con 0x7f0d20073a40 2024-03-28T15:01:08.163 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d27264640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f0d200c8850 con 0x7f0d201516e0 2024-03-28T15:01:08.163 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d25000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0d201516e0 0x7f0d2006c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d25000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0d201516e0 0x7f0d2006c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40660/0 (socket says 172.21.15.191:40660) 2024-03-28T15:01:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d25000640 1 -- 172.21.15.191:0/1777613073 learned_addr learned my addr 172.21.15.191:0/1777613073 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d25801640 1 --2- 172.21.15.191:0/1777613073 >> v2:172.21.15.186:3300/0 conn(0x7f0d20073a40 0x7f0d20157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d25000640 1 -- 172.21.15.191:0/1777613073 >> v2:172.21.15.186:3300/0 conn(0x7f0d20073a40 msgr2=0x7f0d20157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d25000640 1 --2- 172.21.15.191:0/1777613073 >> v2:172.21.15.186:3300/0 conn(0x7f0d20073a40 0x7f0d20157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d25000640 1 -- 172.21.15.191:0/1777613073 >> v1:172.21.15.191:6789/0 conn(0x7f0d200c79a0 legacy=0x7f0d2006e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d25000640 1 -- 172.21.15.191:0/1777613073 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d200c8b30 con 0x7f0d201516e0 2024-03-28T15:01:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.161+0000 7f0d25801640 1 --2- 172.21.15.191:0/1777613073 >> v2:172.21.15.186:3300/0 conn(0x7f0d20073a40 0x7f0d20157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.162+0000 7f0d25000640 1 --2- 172.21.15.191:0/1777613073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d201516e0 0x7f0d2006c8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0d200c84a0 tx=0x7f0d1402e0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c7703a6a146e8b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.162+0000 7f0d1f7fe640 1 -- 172.21.15.191:0/1777613073 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1403b070 con 0x7f0d201516e0 2024-03-28T15:01:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.162+0000 7f0d1f7fe640 1 -- 172.21.15.191:0/1777613073 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d14036440 con 0x7f0d201516e0 2024-03-28T15:01:08.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.162+0000 7f0d1f7fe640 1 -- 172.21.15.191:0/1777613073 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d14036810 con 0x7f0d201516e0 2024-03-28T15:01:08.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.163+0000 7f0d27264640 1 -- 172.21.15.191:0/1777613073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d201516e0 msgr2=0x7f0d2006c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.163+0000 7f0d27264640 1 --2- 172.21.15.191:0/1777613073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d201516e0 0x7f0d2006c8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0d200c84a0 tx=0x7f0d1402e0e0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.163+0000 7f0d27264640 1 -- 172.21.15.191:0/1777613073 shutdown_connections 2024-03-28T15:01:08.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.163+0000 7f0d27264640 1 --2- 172.21.15.191:0/1777613073 >> v2:172.21.15.186:3300/0 conn(0x7f0d20073a40 0x7f0d20157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.163+0000 7f0d27264640 1 --2- 172.21.15.191:0/1777613073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d201516e0 0x7f0d2006c8a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.163+0000 7f0d27264640 1 -- 172.21.15.191:0/1777613073 >> 172.21.15.191:0/1777613073 conn(0x7f0d200cdba0 msgr2=0x7f0d200c9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:08.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.163+0000 7f0d27264640 1 -- 172.21.15.191:0/1777613073 shutdown_connections 2024-03-28T15:01:08.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.163+0000 7f0d27264640 1 -- 172.21.15.191:0/1777613073 wait complete. 2024-03-28T15:01:08.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.164+0000 7f0d27264640 1 Processor -- start 2024-03-28T15:01:08.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.164+0000 7f0d27264640 1 -- start start 2024-03-28T15:01:08.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d27264640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d20073a40 0x7f0d20159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d27264640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d200c79a0 0x7f0d201598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d27264640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d201516e0 0x7f0d201418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d27264640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20151ac0 con 0x7f0d20073a40 2024-03-28T15:01:08.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d27264640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20157590 con 0x7f0d201516e0 2024-03-28T15:01:08.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d27264640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f0d2006ec40 con 0x7f0d200c79a0 2024-03-28T15:01:08.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d25801640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d201516e0 0x7f0d201418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d1ffff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d200c79a0 0x7f0d201598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d25801640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d201516e0 0x7f0d201418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40676/0 (socket says 172.21.15.191:40676) 2024-03-28T15:01:08.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d1ffff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d200c79a0 0x7f0d201598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:33332/0 (socket says 172.21.15.191:33332) 2024-03-28T15:01:08.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d25801640 1 -- 172.21.15.191:0/3115160351 learned_addr learned my addr 172.21.15.191:0/3115160351 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:08.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d25000640 1 --2- 172.21.15.191:0/3115160351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d20073a40 0x7f0d20159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d25801640 1 -- 172.21.15.191:0/3115160351 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d200c79a0 msgr2=0x7f0d201598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d25801640 1 --2- 172.21.15.191:0/3115160351 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d200c79a0 0x7f0d201598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d25801640 1 -- 172.21.15.191:0/3115160351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d20073a40 msgr2=0x7f0d20159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d25801640 1 --2- 172.21.15.191:0/3115160351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d20073a40 0x7f0d20159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d25801640 1 -- 172.21.15.191:0/3115160351 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d201586e0 con 0x7f0d201516e0 2024-03-28T15:01:08.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.165+0000 7f0d1ffff640 1 --2- 172.21.15.191:0/3115160351 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d200c79a0 0x7f0d201598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:08.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.166+0000 7f0d25801640 1 --2- 172.21.15.191:0/3115160351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d201516e0 0x7f0d201418c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0d18007e50 tx=0x7f0d1800d4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:08.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.166+0000 7f0d1d7fa640 1 -- 172.21.15.191:0/3115160351 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1801a070 con 0x7f0d201516e0 2024-03-28T15:01:08.170 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.166+0000 7f0d1d7fa640 1 -- 172.21.15.191:0/3115160351 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d18003990 con 0x7f0d201516e0 2024-03-28T15:01:08.170 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.166+0000 7f0d1d7fa640 1 -- 172.21.15.191:0/3115160351 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d18003d60 con 0x7f0d201516e0 2024-03-28T15:01:08.170 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.166+0000 7f0d27264640 1 -- 172.21.15.191:0/3115160351 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d201f0820 con 0x7f0d201516e0 2024-03-28T15:01:08.170 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.166+0000 7f0d27264640 1 -- 172.21.15.191:0/3115160351 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d20158c60 con 0x7f0d201516e0 2024-03-28T15:01:08.170 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.166+0000 7f0d1d7fa640 1 -- 172.21.15.191:0/3115160351 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0d1801d070 con 0x7f0d201516e0 2024-03-28T15:01:08.170 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.166+0000 7f0d1d7fa640 1 -- 172.21.15.191:0/3115160351 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f0d1801fbe0 con 0x7f0d201516e0 2024-03-28T15:01:08.170 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.167+0000 7f0d27264640 1 -- 172.21.15.191:0/3115160351 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d200686d0 con 0x7f0d201516e0 2024-03-28T15:01:08.171 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.169+0000 7f0d1d7fa640 1 -- 172.21.15.191:0/3115160351 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0d18024020 con 0x7f0d201516e0 2024-03-28T15:01:08.240 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.240+0000 7f0d27264640 1 -- 172.21.15.191:0/3115160351 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "67303d31-dc97-42d2-98cf-a787593dadff", "id": 1} v 0) -- 0x7f0d200723a0 con 0x7f0d201516e0 2024-03-28T15:01:08.246 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.246+0000 7f0d1d7fa640 1 -- 172.21.15.191:0/3115160351 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "67303d31-dc97-42d2-98cf-a787593dadff", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f0d18010070 con 0x7f0d201516e0 2024-03-28T15:01:08.246 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-03-28T15:01:08.247 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.247+0000 7f0d27264640 1 -- 172.21.15.191:0/3115160351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d201516e0 msgr2=0x7f0d201418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.247 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.247+0000 7f0d27264640 1 --2- 172.21.15.191:0/3115160351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d201516e0 0x7f0d201418c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0d18007e50 tx=0x7f0d1800d4a0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.247 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.247+0000 7f0d27264640 1 -- 172.21.15.191:0/3115160351 shutdown_connections 2024-03-28T15:01:08.247 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.247+0000 7f0d27264640 1 --2- 172.21.15.191:0/3115160351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d201516e0 0x7f0d201418c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.247 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.247+0000 7f0d27264640 1 --2- 172.21.15.191:0/3115160351 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d200c79a0 0x7f0d201598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.247 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.247+0000 7f0d27264640 1 --2- 172.21.15.191:0/3115160351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d20073a40 0x7f0d20159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.248 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.247+0000 7f0d27264640 1 -- 172.21.15.191:0/3115160351 >> 172.21.15.191:0/3115160351 conn(0x7f0d200cdba0 msgr2=0x7f0d200c3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:08.248 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.247+0000 7f0d27264640 1 -- 172.21.15.191:0/3115160351 shutdown_connections 2024-03-28T15:01:08.248 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.247+0000 7f0d27264640 1 -- 172.21.15.191:0/3115160351 wait complete. 2024-03-28T15:01:08.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new d34a1208-ef68-48c2-9d0c-7bff414fd34b 2 2024-03-28T15:01:08.306 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:08.306+0000 7fb636a3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:01:08.357 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.358+0000 7f1f9cfb6640 1 Processor -- start 2024-03-28T15:01:08.358 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.358+0000 7f1f9cfb6640 1 -- start start 2024-03-28T15:01:08.359 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.358+0000 7f1f9cfb6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1f981516e0 0x7f1f9806c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.359 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.358+0000 7f1f9cfb6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1f98073a40 0x7f1f98157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.359 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.359+0000 7f1f9cfb6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f980c8570 con 0x7f1f980c79a0 2024-03-28T15:01:08.359 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.359+0000 7f1f9cfb6640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f1f980c8320 con 0x7f1f981516e0 2024-03-28T15:01:08.359 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.359+0000 7f1f9cfb6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f1f980c8850 con 0x7f1f98073a40 2024-03-28T15:01:08.359 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.359+0000 7f1f96d9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1f98073a40 0x7f1f98157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.359 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.359+0000 7f1f96d9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1f98073a40 0x7f1f98157ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40682/0 (socket says 172.21.15.191:40682) 2024-03-28T15:01:08.360 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.359+0000 7f1f96d9d640 1 -- 172.21.15.191:0/2652908074 learned_addr learned my addr 172.21.15.191:0/2652908074 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:08.360 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.359+0000 7f1f9659c640 1 --2- 172.21.15.191:0/2652908074 >> v2:172.21.15.186:3300/0 conn(0x7f1f981516e0 0x7f1f9806c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.360 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.359+0000 7f1f96d9d640 1 -- 172.21.15.191:0/2652908074 >> v2:172.21.15.186:3300/0 conn(0x7f1f981516e0 msgr2=0x7f1f9806c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:08.360 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.359+0000 7f1f96d9d640 1 --2- 172.21.15.191:0/2652908074 >> v2:172.21.15.186:3300/0 conn(0x7f1f981516e0 0x7f1f9806c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.360 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.359+0000 7f1f96d9d640 1 -- 172.21.15.191:0/2652908074 >> v1:172.21.15.186:6789/0 conn(0x7f1f980c79a0 legacy=0x7f1f9806e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.360 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.359+0000 7f1f96d9d640 1 -- 172.21.15.191:0/2652908074 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f980c8b30 con 0x7f1f98073a40 2024-03-28T15:01:08.360 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.360+0000 7f1f96d9d640 1 --2- 172.21.15.191:0/2652908074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f98073a40 0x7f1f98157ab0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1f980c84a0 tx=0x7f1f9002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5ab02d8a5e6f754 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:08.361 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.360+0000 7f1f9559a640 1 -- 172.21.15.191:0/2652908074 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9002ca70 con 0x7f1f98073a40 2024-03-28T15:01:08.361 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.360+0000 7f1f9559a640 1 -- 172.21.15.191:0/2652908074 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f9002cc10 con 0x7f1f98073a40 2024-03-28T15:01:08.361 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.360+0000 7f1f9559a640 1 -- 172.21.15.191:0/2652908074 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f900343f0 con 0x7f1f98073a40 2024-03-28T15:01:08.361 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.361+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/2652908074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f98073a40 msgr2=0x7f1f98157ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.361 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.361+0000 7f1f9cfb6640 1 --2- 172.21.15.191:0/2652908074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f98073a40 0x7f1f98157ab0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1f980c84a0 tx=0x7f1f9002bed0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.362 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.361+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/2652908074 shutdown_connections 2024-03-28T15:01:08.362 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.361+0000 7f1f9cfb6640 1 --2- 172.21.15.191:0/2652908074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f98073a40 0x7f1f98157ab0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.362 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.361+0000 7f1f9cfb6640 1 --2- 172.21.15.191:0/2652908074 >> v2:172.21.15.186:3300/0 conn(0x7f1f981516e0 0x7f1f9806c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.362 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.361+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/2652908074 >> 172.21.15.191:0/2652908074 conn(0x7f1f980cdba0 msgr2=0x7f1f980c9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:08.362 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.361+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/2652908074 shutdown_connections 2024-03-28T15:01:08.362 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.361+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/2652908074 wait complete. 2024-03-28T15:01:08.362 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.362+0000 7f1f9cfb6640 1 Processor -- start 2024-03-28T15:01:08.363 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.362+0000 7f1f9cfb6640 1 -- start start 2024-03-28T15:01:08.363 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:08.362+0000 7fb636a3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:01:08.365 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9cfb6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f98073a40 0x7f1f981f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.365 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9cfb6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f980c79a0 0x7f1f980674f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.365 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9cfb6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f981516e0 0x7f1f980cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.365 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9cfb6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1f98151ac0 con 0x7f1f980c79a0 2024-03-28T15:01:08.365 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9cfb6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1f98157590 con 0x7f1f98073a40 2024-03-28T15:01:08.365 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9cfb6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f1f9806ec40 con 0x7f1f981516e0 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9659c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f98073a40 0x7f1f981f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9659c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f98073a40 0x7f1f981f49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40692/0 (socket says 172.21.15.191:40692) 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f96d9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f981516e0 0x7f1f980cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9659c640 1 -- 172.21.15.191:0/3119266882 learned_addr learned my addr 172.21.15.191:0/3119266882 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f95d9b640 1 --2- 172.21.15.191:0/3119266882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f980c79a0 0x7f1f980674f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9659c640 1 -- 172.21.15.191:0/3119266882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f981516e0 msgr2=0x7f1f980cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9659c640 1 --2- 172.21.15.191:0/3119266882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f981516e0 0x7f1f980cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9659c640 1 -- 172.21.15.191:0/3119266882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f980c79a0 msgr2=0x7f1f980674f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9659c640 1 --2- 172.21.15.191:0/3119266882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f980c79a0 0x7f1f980674f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9659c640 1 -- 172.21.15.191:0/3119266882 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f981494d0 con 0x7f1f98073a40 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f96d9d640 1 --2- 172.21.15.191:0/3119266882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f981516e0 0x7f1f980cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.363+0000 7f1f9659c640 1 --2- 172.21.15.191:0/3119266882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f98073a40 0x7f1f981f49d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1f8800e9b0 tx=0x7f1f8800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.364+0000 7f1f86ffd640 1 -- 172.21.15.191:0/3119266882 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f88017070 con 0x7f1f98073a40 2024-03-28T15:01:08.367 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.364+0000 7f1f86ffd640 1 -- 172.21.15.191:0/3119266882 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f8800c9a0 con 0x7f1f98073a40 2024-03-28T15:01:08.367 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.364+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/3119266882 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f98149770 con 0x7f1f98073a40 2024-03-28T15:01:08.367 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.364+0000 7f1f86ffd640 1 -- 172.21.15.191:0/3119266882 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8800cd90 con 0x7f1f98073a40 2024-03-28T15:01:08.367 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.364+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/3119266882 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f981f41d0 con 0x7f1f98073a40 2024-03-28T15:01:08.367 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.364+0000 7f1f86ffd640 1 -- 172.21.15.191:0/3119266882 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1f8801a070 con 0x7f1f98073a40 2024-03-28T15:01:08.367 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.364+0000 7f1f86ffd640 1 -- 172.21.15.191:0/3119266882 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f1f88006660 con 0x7f1f98073a40 2024-03-28T15:01:08.367 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.365+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/3119266882 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f980686d0 con 0x7f1f98073a40 2024-03-28T15:01:08.367 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.367+0000 7f1f86ffd640 1 -- 172.21.15.191:0/3119266882 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1f88006020 con 0x7f1f98073a40 2024-03-28T15:01:08.409 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:08.409+0000 7fb636a3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:01:08.425 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:08.425+0000 7fed68a42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:01:08.438 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.438+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/3119266882 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d34a1208-ef68-48c2-9d0c-7bff414fd34b", "id": 2} v 0) -- 0x7f1f98147760 con 0x7f1f98073a40 2024-03-28T15:01:08.441 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.442+0000 7f1f86ffd640 1 -- 172.21.15.191:0/3119266882 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d34a1208-ef68-48c2-9d0c-7bff414fd34b", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f1f88005990 con 0x7f1f98073a40 2024-03-28T15:01:08.441 INFO:teuthology.orchestra.run.smithi191.stdout:2 2024-03-28T15:01:08.442 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.442+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/3119266882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f98073a40 msgr2=0x7f1f981f49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.442+0000 7f1f9cfb6640 1 --2- 172.21.15.191:0/3119266882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f98073a40 0x7f1f981f49d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1f8800e9b0 tx=0x7f1f8800ee70 comp rx=0 tx=0).stop 2024-03-28T15:01:08.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.443+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/3119266882 shutdown_connections 2024-03-28T15:01:08.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.443+0000 7f1f9cfb6640 1 --2- 172.21.15.191:0/3119266882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f981516e0 0x7f1f980cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.443+0000 7f1f9cfb6640 1 --2- 172.21.15.191:0/3119266882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f980c79a0 0x7f1f980674f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.443+0000 7f1f9cfb6640 1 --2- 172.21.15.191:0/3119266882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f98073a40 0x7f1f981f49d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.443+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/3119266882 >> 172.21.15.191:0/3119266882 conn(0x7f1f980cdba0 msgr2=0x7f1f9806e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:08.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.443+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/3119266882 shutdown_connections 2024-03-28T15:01:08.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.443+0000 7f1f9cfb6640 1 -- 172.21.15.191:0/3119266882 wait complete. 2024-03-28T15:01:08.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new eed084b7-2749-47d1-acb3-aa2a0ff91646 3 2024-03-28T15:01:08.481 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:08.482+0000 7fb636a3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:01:08.526 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:08.527+0000 7fb636a3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:01:08.556 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.556+0000 7f17b5a47640 1 Processor -- start 2024-03-28T15:01:08.556 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.556+0000 7f17b5a47640 1 -- start start 2024-03-28T15:01:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.557+0000 7f17b5a47640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f17b0154a10 0x7f17b00cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.557+0000 7f17b5a47640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b015de90 con 0x7f17b015b2c0 2024-03-28T15:01:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.557+0000 7f17b5a47640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b00cb6d0 con 0x7f17b00cce90 2024-03-28T15:01:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.557+0000 7f17b5a47640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f17b00c5ae0 con 0x7f17b0154a10 2024-03-28T15:01:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.557+0000 7f17aeffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f17b0154a10 0x7f17b00cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.557+0000 7f17aeffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f17b0154a10 0x7f17b00cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40700/0 (socket says 172.21.15.191:40700) 2024-03-28T15:01:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.557+0000 7f17aeffd640 1 -- 172.21.15.191:0/1371738321 learned_addr learned my addr 172.21.15.191:0/1371738321 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.557+0000 7f17aeffd640 1 -- 172.21.15.191:0/1371738321 >> v1:172.21.15.191:6789/0 conn(0x7f17b00cce90 legacy=0x7f17b0165180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.557+0000 7f17aeffd640 1 -- 172.21.15.191:0/1371738321 >> v1:172.21.15.186:6789/0 conn(0x7f17b015b2c0 legacy=0x7f17b00c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.557+0000 7f17aeffd640 1 -- 172.21.15.191:0/1371738321 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b015e100 con 0x7f17b0154a10 2024-03-28T15:01:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.558+0000 7f17aeffd640 1 --2- 172.21.15.191:0/1371738321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17b0154a10 0x7f17b00cb0b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f17a4009870 tx=0x7f17a402bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b91de4d6cec0f639 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.558+0000 7f17adffb640 1 -- 172.21.15.191:0/1371738321 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a402d870 con 0x7f17b0154a10 2024-03-28T15:01:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.558+0000 7f17adffb640 1 -- 172.21.15.191:0/1371738321 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a402da10 con 0x7f17b0154a10 2024-03-28T15:01:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.558+0000 7f17adffb640 1 -- 172.21.15.191:0/1371738321 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a402dde0 con 0x7f17b0154a10 2024-03-28T15:01:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.559+0000 7f17b5a47640 1 -- 172.21.15.191:0/1371738321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17b0154a10 msgr2=0x7f17b00cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.559+0000 7f17b5a47640 1 --2- 172.21.15.191:0/1371738321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17b0154a10 0x7f17b00cb0b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f17a4009870 tx=0x7f17a402bab0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.559+0000 7f17b5a47640 1 -- 172.21.15.191:0/1371738321 shutdown_connections 2024-03-28T15:01:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.559+0000 7f17b5a47640 1 --2- 172.21.15.191:0/1371738321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17b0154a10 0x7f17b00cb0b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.559+0000 7f17b5a47640 1 -- 172.21.15.191:0/1371738321 >> 172.21.15.191:0/1371738321 conn(0x7f17b006b640 msgr2=0x7f17b006ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.559+0000 7f17b5a47640 1 -- 172.21.15.191:0/1371738321 shutdown_connections 2024-03-28T15:01:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.559+0000 7f17b5a47640 1 -- 172.21.15.191:0/1371738321 wait complete. 2024-03-28T15:01:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.560+0000 7f17b5a47640 1 Processor -- start 2024-03-28T15:01:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.560+0000 7f17b5a47640 1 -- start start 2024-03-28T15:01:08.560 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:08.560+0000 7fed68a42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:01:08.562 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17b5a47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b00cce90 0x7f17b00c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17b5a47640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17b0154a10 0x7f17b01e6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17b5a47640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17b015b2c0 0x7f17b00c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17b5a47640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17b00cb9b0 con 0x7f17b00cce90 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17b5a47640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f17b015e750 con 0x7f17b015b2c0 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17b5a47640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f17b015b710 con 0x7f17b0154a10 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17af7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17b015b2c0 0x7f17b00c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17ae7fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17b0154a10 0x7f17b01e6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17af7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17b015b2c0 0x7f17b00c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40714/0 (socket says 172.21.15.191:40714) 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17af7fe640 1 -- 172.21.15.191:0/4018237791 learned_addr learned my addr 172.21.15.191:0/4018237791 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17aeffd640 1 --2- 172.21.15.191:0/4018237791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b00cce90 0x7f17b00c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17ae7fc640 1 -- 172.21.15.191:0/4018237791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17b015b2c0 msgr2=0x7f17b00c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17ae7fc640 1 --2- 172.21.15.191:0/4018237791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17b015b2c0 0x7f17b00c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17ae7fc640 1 -- 172.21.15.191:0/4018237791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b00cce90 msgr2=0x7f17b00c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17ae7fc640 1 --2- 172.21.15.191:0/4018237791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b00cce90 0x7f17b00c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17ae7fc640 1 -- 172.21.15.191:0/4018237791 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b00b4550 con 0x7f17b0154a10 2024-03-28T15:01:08.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17af7fe640 1 --2- 172.21.15.191:0/4018237791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17b015b2c0 0x7f17b00c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:08.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f17ae7fc640 1 --2- 172.21.15.191:0/4018237791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17b0154a10 0x7f17b01e6ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f17a000c3c0 tx=0x7f17a000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:08.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.561+0000 7f1797fff640 1 -- 172.21.15.191:0/4018237791 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a00052d0 con 0x7f17b0154a10 2024-03-28T15:01:08.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.562+0000 7f1797fff640 1 -- 172.21.15.191:0/4018237791 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a0005490 con 0x7f17b0154a10 2024-03-28T15:01:08.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.562+0000 7f1797fff640 1 -- 172.21.15.191:0/4018237791 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a0007260 con 0x7f17b0154a10 2024-03-28T15:01:08.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.562+0000 7f17b5a47640 1 -- 172.21.15.191:0/4018237791 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b00cf690 con 0x7f17b0154a10 2024-03-28T15:01:08.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.562+0000 7f17b5a47640 1 -- 172.21.15.191:0/4018237791 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b00cfc10 con 0x7f17b0154a10 2024-03-28T15:01:08.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.562+0000 7f1797fff640 1 -- 172.21.15.191:0/4018237791 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f17a0007400 con 0x7f17b0154a10 2024-03-28T15:01:08.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.562+0000 7f1797fff640 1 -- 172.21.15.191:0/4018237791 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f17a0011590 con 0x7f17b0154a10 2024-03-28T15:01:08.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.562+0000 7f17b5a47640 1 -- 172.21.15.191:0/4018237791 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b0069570 con 0x7f17b0154a10 2024-03-28T15:01:08.565 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.565+0000 7f1797fff640 1 -- 172.21.15.191:0/4018237791 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f17a0003a50 con 0x7f17b0154a10 2024-03-28T15:01:08.636 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.636+0000 7f17b5a47640 1 -- 172.21.15.191:0/4018237791 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "eed084b7-2749-47d1-acb3-aa2a0ff91646", "id": 3} v 0) -- 0x7f17b00cbc60 con 0x7f17b0154a10 2024-03-28T15:01:08.642 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.642+0000 7f1797fff640 1 -- 172.21.15.191:0/4018237791 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eed084b7-2749-47d1-acb3-aa2a0ff91646", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f17a0003cf0 con 0x7f17b0154a10 2024-03-28T15:01:08.642 INFO:teuthology.orchestra.run.smithi191.stdout:3 2024-03-28T15:01:08.643 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.643+0000 7f17b5a47640 1 -- 172.21.15.191:0/4018237791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17b0154a10 msgr2=0x7f17b01e6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.643 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.643+0000 7f17b5a47640 1 --2- 172.21.15.191:0/4018237791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17b0154a10 0x7f17b01e6ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f17a000c3c0 tx=0x7f17a000c880 comp rx=0 tx=0).stop 2024-03-28T15:01:08.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.643+0000 7f17b5a47640 1 -- 172.21.15.191:0/4018237791 shutdown_connections 2024-03-28T15:01:08.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.643+0000 7f17b5a47640 1 --2- 172.21.15.191:0/4018237791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17b015b2c0 0x7f17b00c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.643+0000 7f17b5a47640 1 --2- 172.21.15.191:0/4018237791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17b0154a10 0x7f17b01e6ea0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.643+0000 7f17b5a47640 1 --2- 172.21.15.191:0/4018237791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b00cce90 0x7f17b00c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.643+0000 7f17b5a47640 1 -- 172.21.15.191:0/4018237791 >> 172.21.15.191:0/4018237791 conn(0x7f17b006b640 msgr2=0x7f17b015d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:08.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.643+0000 7f17b5a47640 1 -- 172.21.15.191:0/4018237791 shutdown_connections 2024-03-28T15:01:08.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.643+0000 7f17b5a47640 1 -- 172.21.15.191:0/4018237791 wait complete. 2024-03-28T15:01:08.650 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:08.650+0000 7fb636a3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:01:08.654 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 55807f69-d78a-4784-8ea5-433165104a0f 4 2024-03-28T15:01:08.689 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:08.689+0000 7fed68a42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:01:08.723 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:08.723+0000 7fb636a3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:01:08.735 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:08.735+0000 7fed68a42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:01:08.771 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:08.771+0000 7fb636a3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:01:08.794 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.794+0000 7f676369a640 1 Processor -- start 2024-03-28T15:01:08.794 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.794+0000 7f676369a640 1 -- start start 2024-03-28T15:01:08.796 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f676369a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f675c15b440 0x7f675c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.796 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f676369a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f675c0c8a20 0x7f675c06e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f676369a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f675c0cb2b0 con 0x7f675c0b3180 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f676369a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f675c0713a0 con 0x7f675c0c8a20 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f676369a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f675c06b500 con 0x7f675c15b440 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f6760c35640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f675c15b440 0x7f675c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f6760c35640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f675c15b440 0x7f675c06cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40726/0 (socket says 172.21.15.191:40726) 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f6760c35640 1 -- 172.21.15.191:0/641982631 learned_addr learned my addr 172.21.15.191:0/641982631 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f6761c37640 1 --2- 172.21.15.191:0/641982631 >> v2:172.21.15.186:3300/0 conn(0x7f675c0c8a20 0x7f675c06e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f6760c35640 1 -- 172.21.15.191:0/641982631 >> v2:172.21.15.186:3300/0 conn(0x7f675c0c8a20 msgr2=0x7f675c06e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f6760c35640 1 --2- 172.21.15.191:0/641982631 >> v2:172.21.15.186:3300/0 conn(0x7f675c0c8a20 0x7f675c06e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f6760c35640 1 -- 172.21.15.191:0/641982631 >> v1:172.21.15.186:6789/0 conn(0x7f675c0b3180 legacy=0x7f675c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.796+0000 7f6760c35640 1 -- 172.21.15.191:0/641982631 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675c06b7e0 con 0x7f675c15b440 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.797+0000 7f6760c35640 1 --2- 172.21.15.191:0/641982631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f675c15b440 0x7f675c06cff0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f675c06f310 tx=0x7f675802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=611801ead9ed391f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.797+0000 7f6753fff640 1 -- 172.21.15.191:0/641982631 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675802f850 con 0x7f675c15b440 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.797+0000 7f6753fff640 1 -- 172.21.15.191:0/641982631 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f675802f9f0 con 0x7f675c15b440 2024-03-28T15:01:08.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.797+0000 7f6753fff640 1 -- 172.21.15.191:0/641982631 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675802fdc0 con 0x7f675c15b440 2024-03-28T15:01:08.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.798+0000 7f676369a640 1 -- 172.21.15.191:0/641982631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f675c15b440 msgr2=0x7f675c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.798+0000 7f676369a640 1 --2- 172.21.15.191:0/641982631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f675c15b440 0x7f675c06cff0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f675c06f310 tx=0x7f675802ba30 comp rx=0 tx=0).stop 2024-03-28T15:01:08.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.798+0000 7f676369a640 1 -- 172.21.15.191:0/641982631 shutdown_connections 2024-03-28T15:01:08.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.798+0000 7f676369a640 1 --2- 172.21.15.191:0/641982631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f675c15b440 0x7f675c06cff0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.798+0000 7f676369a640 1 --2- 172.21.15.191:0/641982631 >> v2:172.21.15.186:3300/0 conn(0x7f675c0c8a20 0x7f675c06e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.798+0000 7f676369a640 1 -- 172.21.15.191:0/641982631 >> 172.21.15.191:0/641982631 conn(0x7f675c06fa00 msgr2=0x7f675c073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:08.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.798+0000 7f676369a640 1 -- 172.21.15.191:0/641982631 shutdown_connections 2024-03-28T15:01:08.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.798+0000 7f676369a640 1 -- 172.21.15.191:0/641982631 wait complete. 2024-03-28T15:01:08.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.799+0000 7f676369a640 1 Processor -- start 2024-03-28T15:01:08.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.799+0000 7f676369a640 1 -- start start 2024-03-28T15:01:08.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f676369a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f675c0b3180 0x7f675c0c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f676369a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f675c0c8a20 0x7f675c156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f676369a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f675c0c8040 0x7f675c157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:08.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f676369a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f675c15b8d0 con 0x7f675c0c8a20 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f676369a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f675c0b3560 con 0x7f675c0b3180 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f676369a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f675c0cb6c0 con 0x7f675c0c8040 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6761436640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f675c0b3180 0x7f675c0c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6761c37640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f675c0c8040 0x7f675c157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6761436640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f675c0b3180 0x7f675c0c32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40728/0 (socket says 172.21.15.191:40728) 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6761436640 1 -- 172.21.15.191:0/853993763 learned_addr learned my addr 172.21.15.191:0/853993763 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6760c35640 1 --2- 172.21.15.191:0/853993763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f675c0c8a20 0x7f675c156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6761c37640 1 -- 172.21.15.191:0/853993763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f675c0b3180 msgr2=0x7f675c0c32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6761c37640 1 --2- 172.21.15.191:0/853993763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f675c0b3180 0x7f675c0c32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6761c37640 1 -- 172.21.15.191:0/853993763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f675c0c8a20 msgr2=0x7f675c156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6761c37640 1 --2- 172.21.15.191:0/853993763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f675c0c8a20 0x7f675c156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6761c37640 1 -- 172.21.15.191:0/853993763 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675c1f0dc0 con 0x7f675c0c8040 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6761436640 1 --2- 172.21.15.191:0/853993763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f675c0b3180 0x7f675c0c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6761c37640 1 --2- 172.21.15.191:0/853993763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f675c0c8040 0x7f675c157350 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6754007dc0 tx=0x7f675400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:08.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6751ffb640 1 -- 172.21.15.191:0/853993763 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675400edd0 con 0x7f675c0c8040 2024-03-28T15:01:08.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6751ffb640 1 -- 172.21.15.191:0/853993763 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6754003990 con 0x7f675c0c8040 2024-03-28T15:01:08.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f676369a640 1 -- 172.21.15.191:0/853993763 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f675c1f1080 con 0x7f675c0c8040 2024-03-28T15:01:08.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.800+0000 7f6751ffb640 1 -- 172.21.15.191:0/853993763 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754003d80 con 0x7f675c0c8040 2024-03-28T15:01:08.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.801+0000 7f676369a640 1 -- 172.21.15.191:0/853993763 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f675c1f1640 con 0x7f675c0c8040 2024-03-28T15:01:08.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.801+0000 7f6751ffb640 1 -- 172.21.15.191:0/853993763 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f675401b020 con 0x7f675c0c8040 2024-03-28T15:01:08.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.801+0000 7f6751ffb640 1 -- 172.21.15.191:0/853993763 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f6754014b40 con 0x7f675c0c8040 2024-03-28T15:01:08.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.802+0000 7f676369a640 1 -- 172.21.15.191:0/853993763 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f675c158640 con 0x7f675c0c8040 2024-03-28T15:01:08.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.804+0000 7f6751ffb640 1 -- 172.21.15.191:0/853993763 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f675c158640 con 0x7f675c0c8040 2024-03-28T15:01:08.827 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:08.827+0000 7fb636a3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:01:08.848 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:08.848+0000 7fed68a42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:01:08.875 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:08.876+0000 7fb636a3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:01:08.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.876+0000 7f676369a640 1 -- 172.21.15.191:0/853993763 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "55807f69-d78a-4784-8ea5-433165104a0f", "id": 4} v 0) -- 0x7f675c0686d0 con 0x7f675c0c8040 2024-03-28T15:01:08.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.882+0000 7f6751ffb640 1 -- 172.21.15.191:0/853993763 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "55807f69-d78a-4784-8ea5-433165104a0f", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f6754014d90 con 0x7f675c0c8040 2024-03-28T15:01:08.882 INFO:teuthology.orchestra.run.smithi191.stdout:4 2024-03-28T15:01:08.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.883+0000 7f676369a640 1 -- 172.21.15.191:0/853993763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f675c0c8040 msgr2=0x7f675c157350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:08.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.883+0000 7f676369a640 1 --2- 172.21.15.191:0/853993763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f675c0c8040 0x7f675c157350 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6754007dc0 tx=0x7f675400e4b0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.883+0000 7f676369a640 1 -- 172.21.15.191:0/853993763 shutdown_connections 2024-03-28T15:01:08.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.883+0000 7f676369a640 1 --2- 172.21.15.191:0/853993763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f675c0c8040 0x7f675c157350 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.883+0000 7f676369a640 1 --2- 172.21.15.191:0/853993763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f675c0c8a20 0x7f675c156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.883+0000 7f676369a640 1 --2- 172.21.15.191:0/853993763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f675c0b3180 0x7f675c0c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:08.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.883+0000 7f676369a640 1 -- 172.21.15.191:0/853993763 >> 172.21.15.191:0/853993763 conn(0x7f675c06fa00 msgr2=0x7f675c0c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:08.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.883+0000 7f676369a640 1 -- 172.21.15.191:0/853993763 shutdown_connections 2024-03-28T15:01:08.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:08.883+0000 7f676369a640 1 -- 172.21.15.191:0/853993763 wait complete. 2024-03-28T15:01:08.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new bf1fe600-cbd2-47a1-a47e-349a6c7751ba 5 2024-03-28T15:01:08.943 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:08.943+0000 7fed68a42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:01:09.035 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.035+0000 7f024d608640 1 Processor -- start 2024-03-28T15:01:09.036 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.035+0000 7f024d608640 1 -- start start 2024-03-28T15:01:09.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.036+0000 7f024d608640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02480678d0 0x7f024806c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.036+0000 7f024d608640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02480c9300 con 0x7f02480674f0 2024-03-28T15:01:09.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.036+0000 7f024d608640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02480c95e0 con 0x7f024815dcf0 2024-03-28T15:01:09.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.036+0000 7f024d608640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f024806b460 con 0x7f02480678d0 2024-03-28T15:01:09.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.036+0000 7f02477fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f024815dcf0 0x7f02480c8770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:41816/0 (socket says 172.21.15.191:41816) 2024-03-28T15:01:09.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.036+0000 7f02477fe640 1 -- 172.21.15.191:0/947029647 learned_addr learned my addr 172.21.15.191:0/947029647 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-28T15:01:09.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.036+0000 7f023ffff640 1 --2- 172.21.15.191:0/947029647 >> v2:172.21.15.186:3300/0 conn(0x7f02480678d0 0x7f024806c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.036+0000 7f02467fc640 1 -- 172.21.15.191:0/947029647 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048020227 0 0) 0x7f02480c95e0 con 0x7f024815dcf0 2024-03-28T15:01:09.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.036+0000 7f02467fc640 1 -- 172.21.15.191:0/947029647 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0234003520 con 0x7f024815dcf0 2024-03-28T15:01:09.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.036+0000 7f02467fc640 1 -- 172.21.15.191:0/947029647 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1634602519 0 0) 0x7f02480c9300 con 0x7f02480674f0 2024-03-28T15:01:09.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.037+0000 7f02467fc640 1 -- 172.21.15.191:0/947029647 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02480c95e0 con 0x7f02480674f0 2024-03-28T15:01:09.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.037+0000 7f023ffff640 1 -- 172.21.15.191:0/947029647 >> v1:172.21.15.191:6789/0 conn(0x7f024815dcf0 legacy=0x7f02480c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.037+0000 7f023ffff640 1 -- 172.21.15.191:0/947029647 >> v1:172.21.15.186:6789/0 conn(0x7f02480674f0 legacy=0x7f02481545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.037+0000 7f023ffff640 1 -- 172.21.15.191:0/947029647 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024806b740 con 0x7f02480678d0 2024-03-28T15:01:09.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.037+0000 7f023ffff640 1 --2- 172.21.15.191:0/947029647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02480678d0 0x7f024806c220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f02300097a0 tx=0x7f023002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b326c1d99e691a53 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:09.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.038+0000 7f02467fc640 1 -- 172.21.15.191:0/947029647 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023002f810 con 0x7f02480678d0 2024-03-28T15:01:09.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.038+0000 7f02467fc640 1 -- 172.21.15.191:0/947029647 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f023002f9b0 con 0x7f02480678d0 2024-03-28T15:01:09.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.038+0000 7f02467fc640 1 -- 172.21.15.191:0/947029647 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023002fd80 con 0x7f02480678d0 2024-03-28T15:01:09.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.038+0000 7f024d608640 1 -- 172.21.15.191:0/947029647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02480678d0 msgr2=0x7f024806c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.038+0000 7f024d608640 1 --2- 172.21.15.191:0/947029647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02480678d0 0x7f024806c220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f02300097a0 tx=0x7f023002b9f0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.038+0000 7f024d608640 1 -- 172.21.15.191:0/947029647 shutdown_connections 2024-03-28T15:01:09.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.038+0000 7f024d608640 1 --2- 172.21.15.191:0/947029647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02480678d0 0x7f024806c220 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.038+0000 7f024d608640 1 -- 172.21.15.191:0/947029647 >> 172.21.15.191:0/947029647 conn(0x7f02480c9c80 msgr2=0x7f02480ca060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:09.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.038+0000 7f024d608640 1 -- 172.21.15.191:0/947029647 shutdown_connections 2024-03-28T15:01:09.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.038+0000 7f024d608640 1 -- 172.21.15.191:0/947029647 wait complete. 2024-03-28T15:01:09.042 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.039+0000 7f024d608640 1 Processor -- start 2024-03-28T15:01:09.042 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.039+0000 7f024d608640 1 -- start start 2024-03-28T15:01:09.042 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.040+0000 7f024d608640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02480674f0 0x7f02480cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.040+0000 7f024d608640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02480678d0 0x7f02481f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.040+0000 7f024d608640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f024815dcf0 0x7f024814d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.040+0000 7f024d608640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0248154cf0 con 0x7f02480678d0 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.040+0000 7f024d608640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f024806bd00 con 0x7f02480674f0 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.040+0000 7f024d608640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f0248154e70 con 0x7f024815dcf0 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.040+0000 7f0246ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02480674f0 0x7f02480cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.040+0000 7f02477fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f024815dcf0 0x7f024814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.040+0000 7f0246ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02480674f0 0x7f02480cba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40736/0 (socket says 172.21.15.191:40736) 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.040+0000 7f0246ffd640 1 -- 172.21.15.191:0/2230103658 learned_addr learned my addr 172.21.15.191:0/2230103658 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.040+0000 7f023ffff640 1 --2- 172.21.15.191:0/2230103658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02480678d0 0x7f02481f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f0246ffd640 1 -- 172.21.15.191:0/2230103658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f024815dcf0 msgr2=0x7f024814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f0246ffd640 1 --2- 172.21.15.191:0/2230103658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f024815dcf0 0x7f024814d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f0246ffd640 1 -- 172.21.15.191:0/2230103658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02480678d0 msgr2=0x7f02481f34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.043 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f0246ffd640 1 --2- 172.21.15.191:0/2230103658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02480678d0 0x7f02481f34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f0246ffd640 1 -- 172.21.15.191:0/2230103658 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0248153a70 con 0x7f02480674f0 2024-03-28T15:01:09.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f02477fe640 1 --2- 172.21.15.191:0/2230103658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f024815dcf0 0x7f024814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:09.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f0246ffd640 1 --2- 172.21.15.191:0/2230103658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02480674f0 0x7f02480cba60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f023800c500 tx=0x7f023800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:09.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f023f7fe640 1 -- 172.21.15.191:0/2230103658 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02380052d0 con 0x7f02480674f0 2024-03-28T15:01:09.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f023f7fe640 1 -- 172.21.15.191:0/2230103658 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0238005490 con 0x7f02480674f0 2024-03-28T15:01:09.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f024d608640 1 -- 172.21.15.191:0/2230103658 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0248153d10 con 0x7f02480674f0 2024-03-28T15:01:09.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f023f7fe640 1 -- 172.21.15.191:0/2230103658 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238012870 con 0x7f02480674f0 2024-03-28T15:01:09.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f024d608640 1 -- 172.21.15.191:0/2230103658 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02480c6b10 con 0x7f02480674f0 2024-03-28T15:01:09.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f023f7fe640 1 -- 172.21.15.191:0/2230103658 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0238012a10 con 0x7f02480674f0 2024-03-28T15:01:09.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.041+0000 7f023f7fe640 1 -- 172.21.15.191:0/2230103658 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f023801bbe0 con 0x7f02480674f0 2024-03-28T15:01:09.044 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.042+0000 7f024d608640 1 -- 172.21.15.191:0/2230103658 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02480686d0 con 0x7f02480674f0 2024-03-28T15:01:09.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.045+0000 7f023f7fe640 1 -- 172.21.15.191:0/2230103658 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f023801b620 con 0x7f02480674f0 2024-03-28T15:01:09.098 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:09.098+0000 7fed68a42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:01:09.118 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.119+0000 7f024d608640 1 -- 172.21.15.191:0/2230103658 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bf1fe600-cbd2-47a1-a47e-349a6c7751ba", "id": 5} v 0) -- 0x7f024806c220 con 0x7f02480674f0 2024-03-28T15:01:09.124 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.124+0000 7f023f7fe640 1 -- 172.21.15.191:0/2230103658 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bf1fe600-cbd2-47a1-a47e-349a6c7751ba", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f023801b800 con 0x7f02480674f0 2024-03-28T15:01:09.124 INFO:teuthology.orchestra.run.smithi191.stdout:5 2024-03-28T15:01:09.125 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.125+0000 7f024d608640 1 -- 172.21.15.191:0/2230103658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02480674f0 msgr2=0x7f02480cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.125 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.125+0000 7f024d608640 1 --2- 172.21.15.191:0/2230103658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02480674f0 0x7f02480cba60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f023800c500 tx=0x7f023800c9c0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.125 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.125+0000 7f024d608640 1 -- 172.21.15.191:0/2230103658 shutdown_connections 2024-03-28T15:01:09.125 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.125+0000 7f024d608640 1 --2- 172.21.15.191:0/2230103658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f024815dcf0 0x7f024814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.125 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.125+0000 7f024d608640 1 --2- 172.21.15.191:0/2230103658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02480678d0 0x7f02481f34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.125 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.125+0000 7f024d608640 1 --2- 172.21.15.191:0/2230103658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02480674f0 0x7f02480cba60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.125 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.125+0000 7f024d608640 1 -- 172.21.15.191:0/2230103658 >> 172.21.15.191:0/2230103658 conn(0x7f02480c9c80 msgr2=0x7f02480caf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:09.126 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.125+0000 7f024d608640 1 -- 172.21.15.191:0/2230103658 shutdown_connections 2024-03-28T15:01:09.126 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.125+0000 7f024d608640 1 -- 172.21.15.191:0/2230103658 wait complete. 2024-03-28T15:01:09.136 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 42be8188-1e03-46e1-a6a5-8ac79b24afb9 6 2024-03-28T15:01:09.141 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:09.141+0000 7fed68a42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:01:09.237 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.237+0000 7f8e94fe2640 1 Processor -- start 2024-03-28T15:01:09.237 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.237+0000 7f8e94fe2640 1 -- start start 2024-03-28T15:01:09.238 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e94fe2640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 0x7f8e900c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.238 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e94fe2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8e9006c7c0 0x7f8e9006e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.238 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e94fe2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e90157490 con 0x7f8e9006eb10 2024-03-28T15:01:09.238 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e94fe2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8e9006cbc0 con 0x7f8e9006c7c0 2024-03-28T15:01:09.238 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e94fe2640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8e900728a0 con 0x7f8e90151690 2024-03-28T15:01:09.238 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e8e59c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 0x7f8e900c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.238 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e8ed9d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f8e9006eb10 0x7f8e90073170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:41830/0 (socket says 172.21.15.191:41830) 2024-03-28T15:01:09.239 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e8ed9d640 1 -- 172.21.15.191:0/2334017521 learned_addr learned my addr 172.21.15.191:0/2334017521 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-28T15:01:09.239 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e8dd9b640 1 --2- 172.21.15.191:0/2334017521 >> v2:172.21.15.186:3300/0 conn(0x7f8e9006c7c0 0x7f8e9006e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.239 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e8e59c640 1 -- 172.21.15.191:0/2334017521 >> v2:172.21.15.186:3300/0 conn(0x7f8e9006c7c0 msgr2=0x7f8e9006e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:09.239 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e8e59c640 1 --2- 172.21.15.191:0/2334017521 >> v2:172.21.15.186:3300/0 conn(0x7f8e9006c7c0 0x7f8e9006e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.239 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e8e59c640 1 -- 172.21.15.191:0/2334017521 >> v1:172.21.15.191:6789/0 conn(0x7f8e9006eb10 legacy=0x7f8e90073170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.239 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e8d59a640 1 -- 172.21.15.191:0/2334017521 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1121998421 0 0) 0x7f8e90157490 con 0x7f8e9006eb10 2024-03-28T15:01:09.239 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e8e59c640 1 -- 172.21.15.191:0/2334017521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e90157730 con 0x7f8e90151690 2024-03-28T15:01:09.239 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.238+0000 7f8e8dd9b640 1 --2- 172.21.15.191:0/2334017521 >> v2:172.21.15.186:3300/0 conn(0x7f8e9006c7c0 0x7f8e9006e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:09.240 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.239+0000 7f8e8e59c640 1 --2- 172.21.15.191:0/2334017521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 0x7f8e900c7a10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8e800097a0 tx=0x7f8e8002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d94519c62a2d763c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:09.240 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.239+0000 7f8e8d59a640 1 -- 172.21.15.191:0/2334017521 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80034070 con 0x7f8e90151690 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.239+0000 7f8e8d59a640 1 -- 172.21.15.191:0/2334017521 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e80030610 con 0x7f8e90151690 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.239+0000 7f8e8d59a640 1 -- 172.21.15.191:0/2334017521 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e800309e0 con 0x7f8e90151690 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.240+0000 7f8e94fe2640 1 -- 172.21.15.191:0/2334017521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 msgr2=0x7f8e900c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.240+0000 7f8e94fe2640 1 --2- 172.21.15.191:0/2334017521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 0x7f8e900c7a10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8e800097a0 tx=0x7f8e8002b960 comp rx=0 tx=0).stop 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.240+0000 7f8e94fe2640 1 -- 172.21.15.191:0/2334017521 shutdown_connections 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.240+0000 7f8e94fe2640 1 --2- 172.21.15.191:0/2334017521 >> v2:172.21.15.186:3300/0 conn(0x7f8e9006c7c0 0x7f8e9006e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.240+0000 7f8e94fe2640 1 --2- 172.21.15.191:0/2334017521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 0x7f8e900c7a10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.240+0000 7f8e94fe2640 1 -- 172.21.15.191:0/2334017521 >> 172.21.15.191:0/2334017521 conn(0x7f8e900cd010 msgr2=0x7f8e900cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.240+0000 7f8e94fe2640 1 -- 172.21.15.191:0/2334017521 shutdown_connections 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.240+0000 7f8e94fe2640 1 -- 172.21.15.191:0/2334017521 wait complete. 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.241+0000 7f8e94fe2640 1 Processor -- start 2024-03-28T15:01:09.241 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.241+0000 7f8e94fe2640 1 -- start start 2024-03-28T15:01:09.243 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e94fe2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9006c7c0 0x7f8e900c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.243 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e94fe2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e9006eb10 0x7f8e901ee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e94fe2640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 0x7f8e900674f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e94fe2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e90157fd0 con 0x7f8e9006c7c0 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e94fe2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8e90157cf0 con 0x7f8e9006eb10 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e94fe2640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8e901582b0 con 0x7f8e90151690 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8dd9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e9006eb10 0x7f8e901ee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8ed9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 0x7f8e900674f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8dd9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e9006eb10 0x7f8e901ee8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40738/0 (socket says 172.21.15.191:40738) 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8dd9b640 1 -- 172.21.15.191:0/1596141068 learned_addr learned my addr 172.21.15.191:0/1596141068 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8e59c640 1 --2- 172.21.15.191:0/1596141068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9006c7c0 0x7f8e900c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8dd9b640 1 -- 172.21.15.191:0/1596141068 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 msgr2=0x7f8e900674f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8dd9b640 1 --2- 172.21.15.191:0/1596141068 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 0x7f8e900674f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8dd9b640 1 -- 172.21.15.191:0/1596141068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9006c7c0 msgr2=0x7f8e900c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8dd9b640 1 --2- 172.21.15.191:0/1596141068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9006c7c0 0x7f8e900c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8dd9b640 1 -- 172.21.15.191:0/1596141068 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e901483c0 con 0x7f8e9006eb10 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8ed9d640 1 --2- 172.21.15.191:0/1596141068 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 0x7f8e900674f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8e59c640 1 --2- 172.21.15.191:0/1596141068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9006c7c0 0x7f8e900c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e8dd9b640 1 --2- 172.21.15.191:0/1596141068 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e9006eb10 0x7f8e901ee8e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400e760 tx=0x7f8e8400ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e7effd640 1 -- 172.21.15.191:0/1596141068 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8400f040 con 0x7f8e9006eb10 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e7effd640 1 -- 172.21.15.191:0/1596141068 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e84004230 con 0x7f8e9006eb10 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.242+0000 7f8e7effd640 1 -- 172.21.15.191:0/1596141068 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8400ee70 con 0x7f8e9006eb10 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.243+0000 7f8e94fe2640 1 -- 172.21.15.191:0/1596141068 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e901486a0 con 0x7f8e9006eb10 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.243+0000 7f8e94fe2640 1 -- 172.21.15.191:0/1596141068 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e901ee0e0 con 0x7f8e9006eb10 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.243+0000 7f8e7effd640 1 -- 172.21.15.191:0/1596141068 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8e8400ca60 con 0x7f8e9006eb10 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.243+0000 7f8e7effd640 1 -- 172.21.15.191:0/1596141068 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f8e84013b40 con 0x7f8e9006eb10 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.243+0000 7f8e94fe2640 1 -- 172.21.15.191:0/1596141068 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e900686d0 con 0x7f8e9006eb10 2024-03-28T15:01:09.245 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.245+0000 7f8e7effd640 1 -- 172.21.15.191:0/1596141068 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8e84017020 con 0x7f8e9006eb10 2024-03-28T15:01:09.281 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:09.280+0000 7fb636a3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:01:09.297 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:09.297+0000 7fed68a42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:01:09.318 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.318+0000 7f8e94fe2640 1 -- 172.21.15.191:0/1596141068 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "42be8188-1e03-46e1-a6a5-8ac79b24afb9", "id": 6} v 0) -- 0x7f8e900b2550 con 0x7f8e9006eb10 2024-03-28T15:01:09.326 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.326+0000 7f8e7effd640 1 -- 172.21.15.191:0/1596141068 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "42be8188-1e03-46e1-a6a5-8ac79b24afb9", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f8e84013df0 con 0x7f8e9006eb10 2024-03-28T15:01:09.326 INFO:teuthology.orchestra.run.smithi191.stdout:6 2024-03-28T15:01:09.327 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.327+0000 7f8e94fe2640 1 -- 172.21.15.191:0/1596141068 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e9006eb10 msgr2=0x7f8e901ee8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.327 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.327+0000 7f8e94fe2640 1 --2- 172.21.15.191:0/1596141068 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e9006eb10 0x7f8e901ee8e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400e760 tx=0x7f8e8400ec20 comp rx=0 tx=0).stop 2024-03-28T15:01:09.327 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.327+0000 7f8e94fe2640 1 -- 172.21.15.191:0/1596141068 shutdown_connections 2024-03-28T15:01:09.327 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.327+0000 7f8e94fe2640 1 --2- 172.21.15.191:0/1596141068 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e90151690 0x7f8e900674f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.327 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.327+0000 7f8e94fe2640 1 --2- 172.21.15.191:0/1596141068 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e9006eb10 0x7f8e901ee8e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.328 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.327+0000 7f8e94fe2640 1 --2- 172.21.15.191:0/1596141068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9006c7c0 0x7f8e900c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.328 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.327+0000 7f8e94fe2640 1 -- 172.21.15.191:0/1596141068 >> 172.21.15.191:0/1596141068 conn(0x7f8e900cd010 msgr2=0x7f8e9014e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:09.328 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.327+0000 7f8e94fe2640 1 -- 172.21.15.191:0/1596141068 shutdown_connections 2024-03-28T15:01:09.328 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.327+0000 7f8e94fe2640 1 -- 172.21.15.191:0/1596141068 wait complete. 2024-03-28T15:01:09.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new bede2159-0ed9-4214-b776-f184cd0422d5 7 2024-03-28T15:01:09.420 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:09.420+0000 7fb636a3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:01:09.438 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.438+0000 7ff0aaf02640 1 Processor -- start 2024-03-28T15:01:09.438 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.439+0000 7ff0aaf02640 1 -- start start 2024-03-28T15:01:09.439 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.439+0000 7ff0aaf02640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff0a40b3180 0x7ff0a406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.439+0000 7ff0aaf02640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0a40cb2b0 con 0x7ff0a415b440 2024-03-28T15:01:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.439+0000 7ff0aaf02640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0a406b500 con 0x7ff0a40c8a20 2024-03-28T15:01:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.439+0000 7ff0aaf02640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7ff0a40713a0 con 0x7ff0a40b3180 2024-03-28T15:01:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.439+0000 7ff0a8c9e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff0a40b3180 0x7ff0a406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.439+0000 7ff0a8c9e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff0a40b3180 0x7ff0a406cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40754/0 (socket says 172.21.15.191:40754) 2024-03-28T15:01:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.439+0000 7ff0a8c9e640 1 -- 172.21.15.191:0/2032265222 learned_addr learned my addr 172.21.15.191:0/2032265222 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.440+0000 7ff0a8c9e640 1 -- 172.21.15.191:0/2032265222 >> v1:172.21.15.191:6789/0 conn(0x7ff0a40c8a20 legacy=0x7ff0a406e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.440+0000 7ff0a8c9e640 1 -- 172.21.15.191:0/2032265222 >> v1:172.21.15.186:6789/0 conn(0x7ff0a415b440 legacy=0x7ff0a4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.440+0000 7ff0a8c9e640 1 -- 172.21.15.191:0/2032265222 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a406b7e0 con 0x7ff0a40b3180 2024-03-28T15:01:09.441 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.440+0000 7ff0a8c9e640 1 --2- 172.21.15.191:0/2032265222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0a40b3180 0x7ff0a406cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4071520 tx=0x7ff09802bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bee5bfb8bf8a780 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:09.441 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.441+0000 7ff0a37fe640 1 -- 172.21.15.191:0/2032265222 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09802d870 con 0x7ff0a40b3180 2024-03-28T15:01:09.441 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.441+0000 7ff0a37fe640 1 -- 172.21.15.191:0/2032265222 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff09802da10 con 0x7ff0a40b3180 2024-03-28T15:01:09.441 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.441+0000 7ff0a37fe640 1 -- 172.21.15.191:0/2032265222 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09802dde0 con 0x7ff0a40b3180 2024-03-28T15:01:09.442 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.441+0000 7ff0aaf02640 1 -- 172.21.15.191:0/2032265222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0a40b3180 msgr2=0x7ff0a406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.442 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.441+0000 7ff0aaf02640 1 --2- 172.21.15.191:0/2032265222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0a40b3180 0x7ff0a406cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4071520 tx=0x7ff09802bab0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.442 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.441+0000 7ff0aaf02640 1 -- 172.21.15.191:0/2032265222 shutdown_connections 2024-03-28T15:01:09.442 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.441+0000 7ff0aaf02640 1 --2- 172.21.15.191:0/2032265222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0a40b3180 0x7ff0a406cff0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.442 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.441+0000 7ff0aaf02640 1 -- 172.21.15.191:0/2032265222 >> 172.21.15.191:0/2032265222 conn(0x7ff0a406fa00 msgr2=0x7ff0a4073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:09.442 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.441+0000 7ff0aaf02640 1 -- 172.21.15.191:0/2032265222 shutdown_connections 2024-03-28T15:01:09.442 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.441+0000 7ff0aaf02640 1 -- 172.21.15.191:0/2032265222 wait complete. 2024-03-28T15:01:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.442+0000 7ff0aaf02640 1 Processor -- start 2024-03-28T15:01:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.442+0000 7ff0aaf02640 1 -- start start 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.443+0000 7ff0aaf02640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0a40b3180 0x7ff0a415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.443+0000 7ff0aaf02640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0a40c8a20 0x7ff0a4159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.443+0000 7ff0aaf02640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0a4152210 0x7ff0a41f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.443+0000 7ff0aaf02640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a415b8d0 con 0x7ff0a40c8a20 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.443+0000 7ff0aaf02640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a40b3560 con 0x7ff0a40b3180 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.443+0000 7ff0aaf02640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a406ef20 con 0x7ff0a4152210 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.443+0000 7ff0a8c9e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0a40b3180 0x7ff0a415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.443+0000 7ff0a949f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0a4152210 0x7ff0a41f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.443+0000 7ff0a8c9e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0a40b3180 0x7ff0a415ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:40766/0 (socket says 172.21.15.191:40766) 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.443+0000 7ff0a8c9e640 1 -- 172.21.15.191:0/1754849255 learned_addr learned my addr 172.21.15.191:0/1754849255 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.443+0000 7ff0a3fff640 1 --2- 172.21.15.191:0/1754849255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0a40c8a20 0x7ff0a4159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0a8c9e640 1 -- 172.21.15.191:0/1754849255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0a4152210 msgr2=0x7ff0a41f49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0a8c9e640 1 --2- 172.21.15.191:0/1754849255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0a4152210 0x7ff0a41f49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0a8c9e640 1 -- 172.21.15.191:0/1754849255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0a40c8a20 msgr2=0x7ff0a4159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.446 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0a8c9e640 1 --2- 172.21.15.191:0/1754849255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0a40c8a20 0x7ff0a4159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.446 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0a8c9e640 1 -- 172.21.15.191:0/1754849255 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a415a560 con 0x7ff0a40b3180 2024-03-28T15:01:09.446 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0a8c9e640 1 --2- 172.21.15.191:0/1754849255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0a40b3180 0x7ff0a415ac60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff098009960 tx=0x7ff098004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:09.446 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0a17fa640 1 -- 172.21.15.191:0/1754849255 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff098047070 con 0x7ff0a40b3180 2024-03-28T15:01:09.446 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0a17fa640 1 -- 172.21.15.191:0/1754849255 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff098037ac0 con 0x7ff0a40b3180 2024-03-28T15:01:09.446 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0aaf02640 1 -- 172.21.15.191:0/1754849255 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a41e3c90 con 0x7ff0a40b3180 2024-03-28T15:01:09.446 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0a17fa640 1 -- 172.21.15.191:0/1754849255 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09802e440 con 0x7ff0a40b3180 2024-03-28T15:01:09.446 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0aaf02640 1 -- 172.21.15.191:0/1754849255 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a41e39b0 con 0x7ff0a40b3180 2024-03-28T15:01:09.446 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0a17fa640 1 -- 172.21.15.191:0/1754849255 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff09802e5e0 con 0x7ff0a40b3180 2024-03-28T15:01:09.446 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.444+0000 7ff0a17fa640 1 -- 172.21.15.191:0/1754849255 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7ff0980419f0 con 0x7ff0a40b3180 2024-03-28T15:01:09.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.445+0000 7ff0aaf02640 1 -- 172.21.15.191:0/1754849255 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0a40b43c0 con 0x7ff0a40b3180 2024-03-28T15:01:09.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.447+0000 7ff0a17fa640 1 -- 172.21.15.191:0/1754849255 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff0a40b43c0 con 0x7ff0a40b3180 2024-03-28T15:01:09.520 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.520+0000 7ff0aaf02640 1 -- 172.21.15.191:0/1754849255 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bede2159-0ed9-4214-b776-f184cd0422d5", "id": 7} v 0) -- 0x7ff0a4158620 con 0x7ff0a40b3180 2024-03-28T15:01:09.530 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.530+0000 7ff0a17fa640 1 -- 172.21.15.191:0/1754849255 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bede2159-0ed9-4214-b776-f184cd0422d5", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7ff098041c40 con 0x7ff0a40b3180 2024-03-28T15:01:09.530 INFO:teuthology.orchestra.run.smithi191.stdout:7 2024-03-28T15:01:09.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.531+0000 7ff0aaf02640 1 -- 172.21.15.191:0/1754849255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0a40b3180 msgr2=0x7ff0a415ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.531+0000 7ff0aaf02640 1 --2- 172.21.15.191:0/1754849255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0a40b3180 0x7ff0a415ac60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff098009960 tx=0x7ff098004730 comp rx=0 tx=0).stop 2024-03-28T15:01:09.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.531+0000 7ff0aaf02640 1 -- 172.21.15.191:0/1754849255 shutdown_connections 2024-03-28T15:01:09.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.531+0000 7ff0aaf02640 1 --2- 172.21.15.191:0/1754849255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0a4152210 0x7ff0a41f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.531+0000 7ff0aaf02640 1 --2- 172.21.15.191:0/1754849255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0a40c8a20 0x7ff0a4159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.531+0000 7ff0aaf02640 1 --2- 172.21.15.191:0/1754849255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0a40b3180 0x7ff0a415ac60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.531+0000 7ff0aaf02640 1 -- 172.21.15.191:0/1754849255 >> 172.21.15.191:0/1754849255 conn(0x7ff0a406fa00 msgr2=0x7ff0a406cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:09.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.531+0000 7ff0aaf02640 1 -- 172.21.15.191:0/1754849255 shutdown_connections 2024-03-28T15:01:09.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-28T15:01:09.531+0000 7ff0aaf02640 1 -- 172.21.15.191:0/1754849255 wait complete. 2024-03-28T15:01:09.541 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-28T15:01:09.542 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-28T15:01:09.544 INFO:tasks.ceph.osd.0:Started 2024-03-28T15:01:09.544 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-28T15:01:09.544 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-28T15:01:09.545 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:09.545+0000 7fb636a3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:01:09.547 INFO:tasks.ceph.osd.1:Started 2024-03-28T15:01:09.547 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-28T15:01:09.547 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-28T15:01:09.549 INFO:tasks.ceph.osd.2:Started 2024-03-28T15:01:09.549 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-28T15:01:09.549 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-28T15:01:09.551 INFO:tasks.ceph.osd.3:Started 2024-03-28T15:01:09.551 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-28T15:01:09.552 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-28T15:01:09.554 INFO:tasks.ceph.osd.4:Started 2024-03-28T15:01:09.554 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-28T15:01:09.554 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-28T15:01:09.556 INFO:tasks.ceph.osd.5:Started 2024-03-28T15:01:09.556 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-28T15:01:09.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-28T15:01:09.558 INFO:tasks.ceph.osd.6:Started 2024-03-28T15:01:09.558 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-28T15:01:09.558 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-28T15:01:09.561 INFO:tasks.ceph.osd.7:Started 2024-03-28T15:01:09.561 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:09.614 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:09.614+0000 7fb636a3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:01:09.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.694+0000 7f6841b93640 1 Processor -- start 2024-03-28T15:01:09.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.694+0000 7f6841b93640 1 -- start start 2024-03-28T15:01:09.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f6841b93640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f683c1510b0 0x7f683c0cc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f6841b93640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f683c0725b0 0x7f683c0cca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f6841b93640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f683c0c8ad0 0x7f683c1570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f6841b93640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f683c15a410 con 0x7f683c0c8ad0 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f6841b93640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f683c15a6f0 con 0x7f683c0725b0 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f6841b93640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f683c15a9d0 con 0x7f683c1510b0 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f683bfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f683c0c8ad0 0x7f683c1570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f683bfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f683c0c8ad0 0x7f683c1570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:47452/0 (socket says 172.21.15.186:47452) 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f683bfff640 1 -- 172.21.15.186:0/691394281 learned_addr learned my addr 172.21.15.186:0/691394281 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f683b7fe640 1 --2- 172.21.15.186:0/691394281 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f683c1510b0 0x7f683c0cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f6832ffd640 1 --2- 172.21.15.186:0/691394281 >> v2:172.21.15.191:3300/0 conn(0x7f683c0725b0 0x7f683c0cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f683bfff640 1 -- 172.21.15.186:0/691394281 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f683c1510b0 msgr2=0x7f683c0cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f683bfff640 1 --2- 172.21.15.186:0/691394281 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f683c1510b0 0x7f683c0cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f683bfff640 1 -- 172.21.15.186:0/691394281 >> v2:172.21.15.191:3300/0 conn(0x7f683c0725b0 msgr2=0x7f683c0cca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f683bfff640 1 --2- 172.21.15.186:0/691394281 >> v2:172.21.15.191:3300/0 conn(0x7f683c0725b0 0x7f683c0cca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.695+0000 7f683bfff640 1 -- 172.21.15.186:0/691394281 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683c15acb0 con 0x7f683c0c8ad0 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.696+0000 7f683bfff640 1 --2- 172.21.15.186:0/691394281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f683c0c8ad0 0x7f683c1570f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f683c06c520 tx=0x7f683402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=755cd7a4ef318553 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.696+0000 7f683affd640 1 -- 172.21.15.186:0/691394281 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683402ca70 con 0x7f683c0c8ad0 2024-03-28T15:01:09.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.696+0000 7f683affd640 1 -- 172.21.15.186:0/691394281 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f683402cc10 con 0x7f683c0c8ad0 2024-03-28T15:01:09.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.696+0000 7f683affd640 1 -- 172.21.15.186:0/691394281 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68340343f0 con 0x7f683c0c8ad0 2024-03-28T15:01:09.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.697+0000 7f6841b93640 1 -- 172.21.15.186:0/691394281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f683c0c8ad0 msgr2=0x7f683c1570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.697+0000 7f6841b93640 1 --2- 172.21.15.186:0/691394281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f683c0c8ad0 0x7f683c1570f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f683c06c520 tx=0x7f683402bed0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.697+0000 7f6841b93640 1 -- 172.21.15.186:0/691394281 shutdown_connections 2024-03-28T15:01:09.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.697+0000 7f6841b93640 1 --2- 172.21.15.186:0/691394281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f683c0c8ad0 0x7f683c1570f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.697+0000 7f6841b93640 1 --2- 172.21.15.186:0/691394281 >> v2:172.21.15.191:3300/0 conn(0x7f683c0725b0 0x7f683c0cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.697+0000 7f6841b93640 1 --2- 172.21.15.186:0/691394281 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f683c1510b0 0x7f683c0cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.697+0000 7f6841b93640 1 -- 172.21.15.186:0/691394281 >> 172.21.15.186:0/691394281 conn(0x7f683c06f750 msgr2=0x7f683c06fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:09.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.697+0000 7f6841b93640 1 -- 172.21.15.186:0/691394281 shutdown_connections 2024-03-28T15:01:09.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.697+0000 7f6841b93640 1 -- 172.21.15.186:0/691394281 wait complete. 2024-03-28T15:01:09.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.698+0000 7f6841b93640 1 Processor -- start 2024-03-28T15:01:09.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.698+0000 7f6841b93640 1 -- start start 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.699+0000 7f6841b93640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f683c0725b0 0x7f683c1f0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.699+0000 7f6841b93640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f683c0c8ad0 0x7f683c0c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.699+0000 7f6841b93640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f683c1510b0 0x7f683c1ec510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.699+0000 7f6841b93640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f683c0721e0 con 0x7f683c0725b0 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.699+0000 7f6841b93640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f683c072060 con 0x7f683c0c8ad0 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.699+0000 7f6841b93640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f683c072360 con 0x7f683c1510b0 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.699+0000 7f683b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f683c0725b0 0x7f683c1f0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.699+0000 7f683b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f683c0725b0 0x7f683c1f0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:47468/0 (socket says 172.21.15.186:47468) 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.699+0000 7f683b7fe640 1 -- 172.21.15.186:0/3513152579 learned_addr learned my addr 172.21.15.186:0/3513152579 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.699+0000 7f6832ffd640 1 --2- 172.21.15.186:0/3513152579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f683c0c8ad0 0x7f683c0c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.699+0000 7f683bfff640 1 --2- 172.21.15.186:0/3513152579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f683c1510b0 0x7f683c1ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f683b7fe640 1 -- 172.21.15.186:0/3513152579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f683c1510b0 msgr2=0x7f683c1ec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f683b7fe640 1 --2- 172.21.15.186:0/3513152579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f683c1510b0 0x7f683c1ec510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f683b7fe640 1 -- 172.21.15.186:0/3513152579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f683c0c8ad0 msgr2=0x7f683c0c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f683b7fe640 1 --2- 172.21.15.186:0/3513152579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f683c0c8ad0 0x7f683c0c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f683b7fe640 1 -- 172.21.15.186:0/3513152579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683c0c4f70 con 0x7f683c0725b0 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f683b7fe640 1 --2- 172.21.15.186:0/3513152579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f683c0725b0 0x7f683c1f0600 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f682c00dbc0 tx=0x7f682c00b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f6838ff9640 1 -- 172.21.15.186:0/3513152579 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f682c013710 con 0x7f683c0725b0 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f6838ff9640 1 -- 172.21.15.186:0/3513152579 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f682c0138b0 con 0x7f683c0725b0 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f6841b93640 1 -- 172.21.15.186:0/3513152579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f683c0c5250 con 0x7f683c0725b0 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f6838ff9640 1 -- 172.21.15.186:0/3513152579 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f682c013ca0 con 0x7f683c0725b0 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f6841b93640 1 -- 172.21.15.186:0/3513152579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f683c1efb20 con 0x7f683c0725b0 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f6838ff9640 1 -- 172.21.15.186:0/3513152579 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f682c01e400 con 0x7f683c0725b0 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.700+0000 7f6838ff9640 1 -- 172.21.15.186:0/3513152579 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f682c015d00 con 0x7f683c0725b0 2024-03-28T15:01:09.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.701+0000 7f6841b93640 1 -- 172.21.15.186:0/3513152579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f683c0686d0 con 0x7f683c0725b0 2024-03-28T15:01:09.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.703+0000 7f6838ff9640 1 -- 172.21.15.186:0/3513152579 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f682c015630 con 0x7f683c0725b0 2024-03-28T15:01:09.741 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:09.741+0000 7fb636a3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:01:09.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.764+0000 7f6841b93640 1 -- 172.21.15.186:0/3513152579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f683c14b0d0 con 0x7f683c0725b0 2024-03-28T15:01:09.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.765+0000 7f6838ff9640 1 -- 172.21.15.186:0/3513152579 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f682c015810 con 0x7f683c0725b0 2024-03-28T15:01:09.765 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:09.765 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":10,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:09.522002+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:09.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.766+0000 7f6841b93640 1 -- 172.21.15.186:0/3513152579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f683c0725b0 msgr2=0x7f683c1f0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:09.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.766+0000 7f6841b93640 1 --2- 172.21.15.186:0/3513152579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f683c0725b0 0x7f683c1f0600 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f682c00dbc0 tx=0x7f682c00b7a0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.766+0000 7f6841b93640 1 -- 172.21.15.186:0/3513152579 shutdown_connections 2024-03-28T15:01:09.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.766+0000 7f6841b93640 1 --2- 172.21.15.186:0/3513152579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f683c1510b0 0x7f683c1ec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.766+0000 7f6841b93640 1 --2- 172.21.15.186:0/3513152579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f683c0c8ad0 0x7f683c0c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.766+0000 7f6841b93640 1 --2- 172.21.15.186:0/3513152579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f683c0725b0 0x7f683c1f0600 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:09.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.766+0000 7f6841b93640 1 -- 172.21.15.186:0/3513152579 >> 172.21.15.186:0/3513152579 conn(0x7f683c06f750 msgr2=0x7f683c0c7460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:09.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.766+0000 7f6841b93640 1 -- 172.21.15.186:0/3513152579 shutdown_connections 2024-03-28T15:01:09.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:09.766+0000 7f6841b93640 1 -- 172.21.15.186:0/3513152579 wait complete. 2024-03-28T15:01:09.777 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-28T15:01:09.777 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-28T15:01:09.835 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:09.836+0000 7fb636a3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:01:09.992 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:09.992+0000 7fb636a3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:01:10.033 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:10.034+0000 7fb636a3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:01:10.151 INFO:tasks.ceph.osd.1.smithi186.stderr:2024-03-28T15:01:10.151+0000 7fc842393640 -1 Falling back to public interface 2024-03-28T15:01:10.156 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-28T15:01:10.157+0000 7fe0cc0c2640 -1 Falling back to public interface 2024-03-28T15:01:10.160 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-28T15:01:10.160+0000 7f9835810640 -1 Falling back to public interface 2024-03-28T15:01:10.160 INFO:tasks.ceph.osd.2.smithi186.stderr:2024-03-28T15:01:10.160+0000 7f30c4410640 -1 Falling back to public interface 2024-03-28T15:01:10.165 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-28T15:01:10.165+0000 7f31229ff640 -1 Falling back to public interface 2024-03-28T15:01:10.167 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:10.167+0000 7fed68a42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:01:10.168 INFO:tasks.ceph.osd.3.smithi186.stderr:2024-03-28T15:01:10.168+0000 7f445f011640 -1 Falling back to public interface 2024-03-28T15:01:10.172 INFO:tasks.ceph.osd.0.smithi186.stderr:2024-03-28T15:01:10.172+0000 7f2540528640 -1 Falling back to public interface 2024-03-28T15:01:10.176 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-28T15:01:10.177+0000 7f74c9c0e640 -1 Falling back to public interface 2024-03-28T15:01:10.195 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:10.195+0000 7fb636a3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:01:10.444 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:10.444+0000 7fed68a42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:01:10.497 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:10.497+0000 7fed68a42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:01:10.584 DEBUG:teuthology.orchestra.run.smithi186:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:10.612 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:10.612+0000 7fed68a42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:01:10.681 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.681+0000 7f0df196b640 1 Processor -- start 2024-03-28T15:01:10.682 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.681+0000 7f0df196b640 1 -- start start 2024-03-28T15:01:10.683 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0df196b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0dec0c8dd0 0x7f0dec15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:10.683 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0df196b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dec0cbec0 con 0x7f0dec0cafe0 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0df196b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dec0cc1a0 con 0x7f0dec0c77f0 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0df196b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f0dec0c7ec0 con 0x7f0dec0c8dd0 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0deaffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f0dec0cafe0 0x7f0dec0c86c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:45022/0 (socket says 172.21.15.186:45022) 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0deaffd640 1 -- 172.21.15.186:0/197829402 learned_addr learned my addr 172.21.15.186:0/197829402 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0de3fff640 1 --2- 172.21.15.186:0/197829402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0dec0c8dd0 0x7f0dec15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0dea7fc640 1 -- 172.21.15.186:0/197829402 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1928916307 0 0) 0x7f0dec0cbec0 con 0x7f0dec0cafe0 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0dea7fc640 1 -- 172.21.15.186:0/197829402 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dd8003520 con 0x7f0dec0cafe0 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0dea7fc640 1 -- 172.21.15.186:0/197829402 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4240227564 0 0) 0x7f0dec0cc1a0 con 0x7f0dec0c77f0 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0dea7fc640 1 -- 172.21.15.186:0/197829402 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dec0cbec0 con 0x7f0dec0c77f0 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.682+0000 7f0de3fff640 1 -- 172.21.15.186:0/197829402 >> v1:172.21.15.191:6789/0 conn(0x7f0dec0c77f0 legacy=0x7f0dec1651d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.683+0000 7f0de3fff640 1 -- 172.21.15.186:0/197829402 >> v1:172.21.15.186:6789/0 conn(0x7f0dec0cafe0 legacy=0x7f0dec0c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.683+0000 7f0de3fff640 1 -- 172.21.15.186:0/197829402 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dec0c81a0 con 0x7f0dec0c8dd0 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.683+0000 7f0de3fff640 1 --2- 172.21.15.186:0/197829402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0dec0c8dd0 0x7f0dec15e820 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40097c0 tx=0x7f0dd402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=388c6ddab4cf8f9c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:10.684 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.683+0000 7f0dea7fc640 1 -- 172.21.15.186:0/197829402 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4035020 con 0x7f0dec0c8dd0 2024-03-28T15:01:10.685 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.683+0000 7f0dea7fc640 1 -- 172.21.15.186:0/197829402 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd4031890 con 0x7f0dec0c8dd0 2024-03-28T15:01:10.685 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.683+0000 7f0dea7fc640 1 -- 172.21.15.186:0/197829402 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4031c60 con 0x7f0dec0c8dd0 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.684+0000 7f0df196b640 1 -- 172.21.15.186:0/197829402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0dec0c8dd0 msgr2=0x7f0dec15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.684+0000 7f0df196b640 1 --2- 172.21.15.186:0/197829402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0dec0c8dd0 0x7f0dec15e820 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40097c0 tx=0x7f0dd402b930 comp rx=0 tx=0).stop 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.684+0000 7f0df196b640 1 -- 172.21.15.186:0/197829402 shutdown_connections 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.684+0000 7f0df196b640 1 --2- 172.21.15.186:0/197829402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0dec0c8dd0 0x7f0dec15e820 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.684+0000 7f0df196b640 1 -- 172.21.15.186:0/197829402 >> 172.21.15.186:0/197829402 conn(0x7f0dec0d0140 msgr2=0x7f0dec0d0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.684+0000 7f0df196b640 1 -- 172.21.15.186:0/197829402 shutdown_connections 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.684+0000 7f0df196b640 1 -- 172.21.15.186:0/197829402 wait complete. 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.685+0000 7f0df196b640 1 Processor -- start 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.685+0000 7f0df196b640 1 -- start start 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0df196b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec0c77f0 0x7f0dec156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0df196b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dec0c8dd0 0x7f0dec1e6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0df196b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0dec0cafe0 0x7f0dec1e6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:10.688 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0df196b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec0b6090 con 0x7f0dec0c77f0 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0df196b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec0b5f10 con 0x7f0dec0c8dd0 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0df196b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f0dec154e80 con 0x7f0dec0cafe0 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0deaffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec0c77f0 0x7f0dec156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0deaffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec0c77f0 0x7f0dec156830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:47504/0 (socket says 172.21.15.186:47504) 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0deaffd640 1 -- 172.21.15.186:0/3134008100 learned_addr learned my addr 172.21.15.186:0/3134008100 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0deb7fe640 1 --2- 172.21.15.186:0/3134008100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0dec0cafe0 0x7f0dec1e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0de3fff640 1 --2- 172.21.15.186:0/3134008100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dec0c8dd0 0x7f0dec1e6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0deaffd640 1 -- 172.21.15.186:0/3134008100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0dec0cafe0 msgr2=0x7f0dec1e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0deaffd640 1 --2- 172.21.15.186:0/3134008100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0dec0cafe0 0x7f0dec1e6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0deaffd640 1 -- 172.21.15.186:0/3134008100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dec0c8dd0 msgr2=0x7f0dec1e6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0deaffd640 1 --2- 172.21.15.186:0/3134008100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dec0c8dd0 0x7f0dec1e6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0deaffd640 1 -- 172.21.15.186:0/3134008100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dec1e5130 con 0x7f0dec0c77f0 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.686+0000 7f0deaffd640 1 --2- 172.21.15.186:0/3134008100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec0c77f0 0x7f0dec156830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc00c530 tx=0x7f0ddc00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:10.689 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.687+0000 7f0de8ff9640 1 -- 172.21.15.186:0/3134008100 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc017070 con 0x7f0dec0c77f0 2024-03-28T15:01:10.690 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.687+0000 7f0de8ff9640 1 -- 172.21.15.186:0/3134008100 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ddc005ce0 con 0x7f0dec0c77f0 2024-03-28T15:01:10.690 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.687+0000 7f0de8ff9640 1 -- 172.21.15.186:0/3134008100 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc007270 con 0x7f0dec0c77f0 2024-03-28T15:01:10.690 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.687+0000 7f0df196b640 1 -- 172.21.15.186:0/3134008100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dec158690 con 0x7f0dec0c77f0 2024-03-28T15:01:10.690 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.687+0000 7f0df196b640 1 -- 172.21.15.186:0/3134008100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dec0c35c0 con 0x7f0dec0c77f0 2024-03-28T15:01:10.690 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.688+0000 7f0de8ff9640 1 -- 172.21.15.186:0/3134008100 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 2) ==== 45453+0+0 (secure 0 0 0) 0x7f0ddc01a070 con 0x7f0dec0c77f0 2024-03-28T15:01:10.690 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.688+0000 7f0df196b640 1 -- 172.21.15.186:0/3134008100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dec0ca790 con 0x7f0dec0c77f0 2024-03-28T15:01:10.690 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.688+0000 7f0de8ff9640 1 -- 172.21.15.186:0/3134008100 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f0ddc049b30 con 0x7f0dec0c77f0 2024-03-28T15:01:10.690 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.690+0000 7f0de8ff9640 1 -- 172.21.15.186:0/3134008100 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0dec0ca790 con 0x7f0dec0c77f0 2024-03-28T15:01:10.750 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.750+0000 7f0df196b640 1 -- 172.21.15.186:0/3134008100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0dec04f8b0 con 0x7f0dec0c77f0 2024-03-28T15:01:10.751 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.751+0000 7f0de8ff9640 1 -- 172.21.15.186:0/3134008100 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f0ddc007410 con 0x7f0dec0c77f0 2024-03-28T15:01:10.751 INFO:teuthology.misc.health.smithi186.stdout: 2024-03-28T15:01:10.751 INFO:teuthology.misc.health.smithi186.stdout:{"epoch":10,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:09.522002+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:10.752 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.751+0000 7f0df196b640 1 -- 172.21.15.186:0/3134008100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec0c77f0 msgr2=0x7f0dec156830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:10.752 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.751+0000 7f0df196b640 1 --2- 172.21.15.186:0/3134008100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec0c77f0 0x7f0dec156830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc00c530 tx=0x7f0ddc00c9f0 comp rx=0 tx=0).stop 2024-03-28T15:01:10.752 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.751+0000 7f0df196b640 1 -- 172.21.15.186:0/3134008100 shutdown_connections 2024-03-28T15:01:10.752 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.751+0000 7f0df196b640 1 --2- 172.21.15.186:0/3134008100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0dec0cafe0 0x7f0dec1e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:10.752 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.751+0000 7f0df196b640 1 --2- 172.21.15.186:0/3134008100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dec0c8dd0 0x7f0dec1e6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:10.753 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.751+0000 7f0df196b640 1 --2- 172.21.15.186:0/3134008100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec0c77f0 0x7f0dec156830 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:10.753 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.751+0000 7f0df196b640 1 -- 172.21.15.186:0/3134008100 >> 172.21.15.186:0/3134008100 conn(0x7f0dec0d0140 msgr2=0x7f0dec151700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:10.753 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.752+0000 7f0df196b640 1 -- 172.21.15.186:0/3134008100 shutdown_connections 2024-03-28T15:01:10.753 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:10.752+0000 7f0df196b640 1 -- 172.21.15.186:0/3134008100 wait complete. 2024-03-28T15:01:10.761 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T15:01:11.040 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:11.040+0000 7fb636a3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:01:11.319 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:11.319+0000 7fb636a3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:01:11.370 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:11.370+0000 7fb636a3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:01:11.487 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-28T15:01:11.487+0000 7fb636a3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:01:11.637 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:11.637+0000 7fed095eb640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-28T15:01:11.638 INFO:tasks.ceph.mgr.y.smithi186.stderr:2024-03-28T15:01:11.637+0000 7fed095eb640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-28T15:01:13.599 INFO:tasks.ceph.osd.1.smithi186.stderr:2024-03-28T15:01:13.599+0000 7fc842393640 -1 osd.1 0 log_to_monitors true 2024-03-28T15:01:13.653 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-28T15:01:13.652+0000 7f9835810640 -1 osd.5 0 log_to_monitors true 2024-03-28T15:01:13.672 INFO:tasks.ceph.osd.3.smithi186.stderr:2024-03-28T15:01:13.672+0000 7f445f011640 -1 osd.3 0 log_to_monitors true 2024-03-28T15:01:13.682 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-28T15:01:13.681+0000 7f31229ff640 -1 osd.7 0 log_to_monitors true 2024-03-28T15:01:13.692 INFO:tasks.ceph.osd.0.smithi186.stderr:2024-03-28T15:01:13.692+0000 7f2540528640 -1 osd.0 0 log_to_monitors true 2024-03-28T15:01:13.696 INFO:tasks.ceph.osd.2.smithi186.stderr:2024-03-28T15:01:13.696+0000 7f30c4410640 -1 osd.2 0 log_to_monitors true 2024-03-28T15:01:13.702 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-28T15:01:13.702+0000 7fe0cc0c2640 -1 osd.6 0 log_to_monitors true 2024-03-28T15:01:13.707 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-28T15:01:13.706+0000 7f74c9c0e640 -1 osd.4 0 log_to_monitors true 2024-03-28T15:01:17.527 INFO:tasks.ceph.osd.1.smithi186.stderr:2024-03-28T15:01:17.526+0000 7fc83e329640 -1 osd.1 0 waiting for initial osdmap 2024-03-28T15:01:17.540 INFO:tasks.ceph.osd.1.smithi186.stderr:2024-03-28T15:01:17.539+0000 7fc836149640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:01:17.542 INFO:tasks.ceph.osd.0.smithi186.stderr:2024-03-28T15:01:17.542+0000 7f253c4be640 -1 osd.0 0 waiting for initial osdmap 2024-03-28T15:01:17.553 INFO:tasks.ceph.osd.0.smithi186.stderr:2024-03-28T15:01:17.553+0000 7f25342de640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:01:17.554 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-28T15:01:17.554+0000 7f311e995640 -1 osd.7 0 waiting for initial osdmap 2024-03-28T15:01:17.560 INFO:tasks.ceph.osd.3.smithi186.stderr:2024-03-28T15:01:17.560+0000 7f445a9af640 -1 osd.3 0 waiting for initial osdmap 2024-03-28T15:01:17.566 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-28T15:01:17.566+0000 7f31167b5640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:01:17.570 DEBUG:teuthology.orchestra.run.smithi186:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:17.570 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-28T15:01:17.568+0000 7f74c5675640 -1 osd.4 0 waiting for initial osdmap 2024-03-28T15:01:17.571 INFO:tasks.ceph.osd.2.smithi186.stderr:2024-03-28T15:01:17.571+0000 7f30bfde4640 -1 osd.2 0 waiting for initial osdmap 2024-03-28T15:01:17.574 INFO:tasks.ceph.osd.3.smithi186.stderr:2024-03-28T15:01:17.574+0000 7f44527cf640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:01:17.584 INFO:tasks.ceph.osd.2.smithi186.stderr:2024-03-28T15:01:17.584+0000 7f30b7c04640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:01:17.584 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-28T15:01:17.583+0000 7f74bd495640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:01:17.584 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-28T15:01:17.583+0000 7fe0c8058640 -1 osd.6 0 waiting for initial osdmap 2024-03-28T15:01:17.596 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-28T15:01:17.596+0000 7fe0bfe78640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:01:17.598 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-28T15:01:17.598+0000 7f98310c6640 -1 osd.5 0 waiting for initial osdmap 2024-03-28T15:01:17.613 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-28T15:01:17.612+0000 7f9828ee6640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:01:17.660 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.660+0000 7f0047b0f640 1 Processor -- start 2024-03-28T15:01:17.660 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.660+0000 7f0047b0f640 1 -- start start 2024-03-28T15:01:17.661 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f0047b0f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0040157f30 0x7f004015b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:17.661 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f0047b0f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00400c9b40 0x7f004006f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:17.661 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f0047b0f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004006e4c0 con 0x7f00400c8b30 2024-03-28T15:01:17.662 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f0047b0f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f004006cc50 con 0x7f00400c9b40 2024-03-28T15:01:17.662 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f0047b0f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f004006e220 con 0x7f0040157f30 2024-03-28T15:01:17.662 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f00450aa640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00400c9b40 0x7f004006f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:17.662 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f00450aa640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00400c9b40 0x7f004006f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:47552/0 (socket says 172.21.15.186:47552) 2024-03-28T15:01:17.662 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f00450aa640 1 -- 172.21.15.186:0/4049888827 learned_addr learned my addr 172.21.15.186:0/4049888827 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:17.662 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f00450aa640 1 -- 172.21.15.186:0/4049888827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0040157f30 msgr2=0x7f004015b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:17.662 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f00450aa640 1 --2- 172.21.15.186:0/4049888827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0040157f30 0x7f004015b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:17.662 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f00450aa640 1 -- 172.21.15.186:0/4049888827 >> v1:172.21.15.191:6789/0 conn(0x7f00400c8b30 legacy=0x7f004015ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:17.662 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.661+0000 7f00450aa640 1 -- 172.21.15.186:0/4049888827 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004006c690 con 0x7f00400c9b40 2024-03-28T15:01:17.663 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.662+0000 7f00450aa640 1 --2- 172.21.15.186:0/4049888827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00400c9b40 0x7f004006f520 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0030009870 tx=0x7f003002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9879fa797e70fb91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:17.663 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.662+0000 7f00448a9640 1 -- 172.21.15.186:0/4049888827 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0030034070 con 0x7f00400c9b40 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.662+0000 7f00448a9640 1 -- 172.21.15.186:0/4049888827 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0030030840 con 0x7f00400c9b40 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.662+0000 7f00448a9640 1 -- 172.21.15.186:0/4049888827 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0030030c10 con 0x7f00400c9b40 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.662+0000 7f0047b0f640 1 -- 172.21.15.186:0/4049888827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00400c9b40 msgr2=0x7f004006f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.662+0000 7f0047b0f640 1 --2- 172.21.15.186:0/4049888827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00400c9b40 0x7f004006f520 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0030009870 tx=0x7f003002b980 comp rx=0 tx=0).stop 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.662+0000 7f0047b0f640 1 -- 172.21.15.186:0/4049888827 shutdown_connections 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.662+0000 7f0047b0f640 1 --2- 172.21.15.186:0/4049888827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00400c9b40 0x7f004006f520 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.662+0000 7f0047b0f640 1 --2- 172.21.15.186:0/4049888827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0040157f30 0x7f004015b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.662+0000 7f0047b0f640 1 -- 172.21.15.186:0/4049888827 >> 172.21.15.186:0/4049888827 conn(0x7f004006baa0 msgr2=0x7f0040073680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.663+0000 7f0047b0f640 1 -- 172.21.15.186:0/4049888827 shutdown_connections 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.663+0000 7f0047b0f640 1 -- 172.21.15.186:0/4049888827 wait complete. 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.664+0000 7f0047b0f640 1 Processor -- start 2024-03-28T15:01:17.664 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.664+0000 7f0047b0f640 1 -- start start 2024-03-28T15:01:17.667 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.664+0000 7f0047b0f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00400c8b30 0x7f00401f4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:17.667 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.664+0000 7f0047b0f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00400c9b40 0x7f004015a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:17.667 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.664+0000 7f0047b0f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0040157f30 0x7f004014eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:17.667 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.664+0000 7f0047b0f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f00400c52e0 con 0x7f00400c8b30 2024-03-28T15:01:17.667 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.664+0000 7f0047b0f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f004015ab70 con 0x7f00400c9b40 2024-03-28T15:01:17.667 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.664+0000 7f0047b0f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f0040158310 con 0x7f0040157f30 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.664+0000 7f00458ab640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00400c8b30 0x7f00401f4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.664+0000 7f00458ab640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00400c8b30 0x7f00401f4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:47556/0 (socket says 172.21.15.186:47556) 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.664+0000 7f00458ab640 1 -- 172.21.15.186:0/1530262958 learned_addr learned my addr 172.21.15.186:0/1530262958 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f00460ac640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0040157f30 0x7f004014eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f00450aa640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00400c9b40 0x7f004015a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f00458ab640 1 -- 172.21.15.186:0/1530262958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0040157f30 msgr2=0x7f004014eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f00458ab640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0040157f30 0x7f004014eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f00458ab640 1 -- 172.21.15.186:0/1530262958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00400c9b40 msgr2=0x7f004015a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f00458ab640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00400c9b40 0x7f004015a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f00458ab640 1 -- 172.21.15.186:0/1530262958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00401514c0 con 0x7f00400c8b30 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f00458ab640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00400c8b30 0x7f00401f4670 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f003c00ba80 tx=0x7f003c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f00367fc640 1 -- 172.21.15.186:0/1530262958 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c019070 con 0x7f00400c8b30 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f00367fc640 1 -- 172.21.15.186:0/1530262958 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f003c00cc20 con 0x7f00400c8b30 2024-03-28T15:01:17.668 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f0047b0f640 1 -- 172.21.15.186:0/1530262958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00401593f0 con 0x7f00400c8b30 2024-03-28T15:01:17.669 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f00367fc640 1 -- 172.21.15.186:0/1530262958 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c014440 con 0x7f00400c8b30 2024-03-28T15:01:17.669 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.665+0000 7f0047b0f640 1 -- 172.21.15.186:0/1530262958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00400c4470 con 0x7f00400c8b30 2024-03-28T15:01:17.669 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.666+0000 7f0047b0f640 1 -- 172.21.15.186:0/1530262958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f004004f8b0 con 0x7f00400c8b30 2024-03-28T15:01:17.669 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.667+0000 7f00367fc640 1 -- 172.21.15.186:0/1530262958 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90800+0+0 (secure 0 0 0) 0x7f003c01c070 con 0x7f00400c8b30 2024-03-28T15:01:17.669 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.667+0000 7f00367fc640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f002406c4a0 0x7f002406e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:17.669 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.667+0000 7f00450aa640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f002406c4a0 0x7f002406e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:17.669 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.667+0000 7f00367fc640 1 -- 172.21.15.186:0/1530262958 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f003c08a440 con 0x7f00400c8b30 2024-03-28T15:01:17.669 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.667+0000 7f00450aa640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f002406c4a0 0x7f002406e960 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0030036000 tx=0x7f003003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:17.671 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.672+0000 7f00367fc640 1 -- 172.21.15.186:0/1530262958 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f003c059280 con 0x7f00400c8b30 2024-03-28T15:01:17.849 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.849+0000 7f0047b0f640 1 -- 172.21.15.186:0/1530262958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00400c4e40 con 0x7f00400c8b30 2024-03-28T15:01:17.849 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.849+0000 7f00367fc640 1 -- 172.21.15.186:0/1530262958 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f003c05d580 con 0x7f00400c8b30 2024-03-28T15:01:17.850 INFO:teuthology.misc.health.smithi186.stdout: 2024-03-28T15:01:17.850 INFO:teuthology.misc.health.smithi186.stdout:{"epoch":12,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:15.464271+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:17.852 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.851+0000 7f0047b0f640 1 -- 172.21.15.186:0/1530262958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f002406c4a0 msgr2=0x7f002406e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:17.852 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.851+0000 7f0047b0f640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f002406c4a0 0x7f002406e960 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0030036000 tx=0x7f003003b040 comp rx=0 tx=0).stop 2024-03-28T15:01:17.852 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.851+0000 7f0047b0f640 1 -- 172.21.15.186:0/1530262958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00400c8b30 msgr2=0x7f00401f4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:17.852 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.851+0000 7f0047b0f640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00400c8b30 0x7f00401f4670 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f003c00ba80 tx=0x7f003c00bf40 comp rx=0 tx=0).stop 2024-03-28T15:01:17.852 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.851+0000 7f0047b0f640 1 -- 172.21.15.186:0/1530262958 shutdown_connections 2024-03-28T15:01:17.852 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.851+0000 7f0047b0f640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f002406c4a0 0x7f002406e960 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:17.852 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.851+0000 7f0047b0f640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0040157f30 0x7f004014eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:17.852 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.851+0000 7f0047b0f640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00400c9b40 0x7f004015a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:17.852 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.852+0000 7f0047b0f640 1 --2- 172.21.15.186:0/1530262958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00400c8b30 0x7f00401f4670 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:17.853 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.852+0000 7f0047b0f640 1 -- 172.21.15.186:0/1530262958 >> 172.21.15.186:0/1530262958 conn(0x7f004006baa0 msgr2=0x7f0040073680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:17.853 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.852+0000 7f0047b0f640 1 -- 172.21.15.186:0/1530262958 shutdown_connections 2024-03-28T15:01:17.853 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:17.852+0000 7f0047b0f640 1 -- 172.21.15.186:0/1530262958 wait complete. 2024-03-28T15:01:17.861 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T15:01:24.668 DEBUG:teuthology.orchestra.run.smithi186:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:24.767 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.767+0000 7fbf5b615640 1 Processor -- start 2024-03-28T15:01:24.767 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.767+0000 7fbf5b615640 1 -- start start 2024-03-28T15:01:24.769 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.768+0000 7fbf5b615640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbf5406c7f0 0x7fbf5415aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:24.769 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.768+0000 7fbf5b615640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf5406cbd0 0x7fbf541548d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:24.769 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.768+0000 7fbf5b615640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf540cc970 con 0x7fbf540c7ed0 2024-03-28T15:01:24.769 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.768+0000 7fbf5b615640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fbf5415ef10 con 0x7fbf5406c7f0 2024-03-28T15:01:24.769 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.768+0000 7fbf5b615640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fbf540b6020 con 0x7fbf5406cbd0 2024-03-28T15:01:24.769 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.768+0000 7fbf58bb0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf5406cbd0 0x7fbf541548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:24.769 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.768+0000 7fbf58bb0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf5406cbd0 0x7fbf541548d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:49436/0 (socket says 172.21.15.186:49436) 2024-03-28T15:01:24.769 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.768+0000 7fbf58bb0640 1 -- 172.21.15.186:0/3937553747 learned_addr learned my addr 172.21.15.186:0/3937553747 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:24.769 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.768+0000 7fbf59bb2640 1 --1- 172.21.15.186:0/3937553747 >> v1:172.21.15.191:6789/0 conn(0x7fbf540c7ed0 0x7fbf540c83e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:39472/0 (socket says 172.21.15.186:39472) 2024-03-28T15:01:24.770 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.768+0000 7fbf593b1640 1 --2- 172.21.15.186:0/3937553747 >> v2:172.21.15.191:3300/0 conn(0x7fbf5406c7f0 0x7fbf5415aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:24.770 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.769+0000 7fbf4bfff640 1 -- 172.21.15.186:0/3937553747 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517215667 0 0) 0x7fbf540cc970 con 0x7fbf540c7ed0 2024-03-28T15:01:24.770 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.769+0000 7fbf4bfff640 1 -- 172.21.15.186:0/3937553747 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf40003520 con 0x7fbf540c7ed0 2024-03-28T15:01:24.770 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.769+0000 7fbf593b1640 1 -- 172.21.15.186:0/3937553747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf5406cbd0 msgr2=0x7fbf541548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:24.770 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.769+0000 7fbf593b1640 1 --2- 172.21.15.186:0/3937553747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf5406cbd0 0x7fbf541548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:24.770 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.769+0000 7fbf593b1640 1 -- 172.21.15.186:0/3937553747 >> v1:172.21.15.191:6789/0 conn(0x7fbf540c7ed0 legacy=0x7fbf540c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:24.770 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.769+0000 7fbf593b1640 1 -- 172.21.15.186:0/3937553747 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf540cc6b0 con 0x7fbf5406c7f0 2024-03-28T15:01:24.770 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.769+0000 7fbf58bb0640 1 --2- 172.21.15.186:0/3937553747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf5406cbd0 0x7fbf541548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:24.771 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf593b1640 1 --2- 172.21.15.186:0/3937553747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf5406c7f0 0x7fbf5415aad0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbf500097a0 tx=0x7fbf5002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c28cb7bcbeb5ea09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:24.771 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf4bfff640 1 -- 172.21.15.186:0/3937553747 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf50034070 con 0x7fbf5406c7f0 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf4bfff640 1 -- 172.21.15.186:0/3937553747 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf50030610 con 0x7fbf5406c7f0 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf4bfff640 1 -- 172.21.15.186:0/3937553747 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf500309c0 con 0x7fbf5406c7f0 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf5b615640 1 -- 172.21.15.186:0/3937553747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf5406c7f0 msgr2=0x7fbf5415aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf5b615640 1 --2- 172.21.15.186:0/3937553747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf5406c7f0 0x7fbf5415aad0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbf500097a0 tx=0x7fbf5002b960 comp rx=0 tx=0).stop 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf5b615640 1 -- 172.21.15.186:0/3937553747 shutdown_connections 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf5b615640 1 --2- 172.21.15.186:0/3937553747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf5406cbd0 0x7fbf541548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf5b615640 1 --2- 172.21.15.186:0/3937553747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf5406c7f0 0x7fbf5415aad0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf5b615640 1 -- 172.21.15.186:0/3937553747 >> 172.21.15.186:0/3937553747 conn(0x7fbf540ccb40 msgr2=0x7fbf540ccf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf5b615640 1 -- 172.21.15.186:0/3937553747 shutdown_connections 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.770+0000 7fbf5b615640 1 -- 172.21.15.186:0/3937553747 wait complete. 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf5b615640 1 Processor -- start 2024-03-28T15:01:24.772 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf5b615640 1 -- start start 2024-03-28T15:01:24.774 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf5b615640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf5406c7f0 0x7fbf54153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:24.774 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf5b615640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf5406cbd0 0x7fbf541f03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:24.774 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf5b615640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf540c7ed0 0x7fbf541f0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:24.774 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf5b615640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5415e710 con 0x7fbf5406cbd0 2024-03-28T15:01:24.774 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf5b615640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5415e430 con 0x7fbf5406c7f0 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf5b615640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fbf5415e9f0 con 0x7fbf540c7ed0 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf58bb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf5406cbd0 0x7fbf541f03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf58bb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf5406cbd0 0x7fbf541f03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40108/0 (socket says 172.21.15.186:40108) 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf58bb0640 1 -- 172.21.15.186:0/800921416 learned_addr learned my addr 172.21.15.186:0/800921416 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf59bb2640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf540c7ed0 0x7fbf541f0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.772+0000 7fbf593b1640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf5406c7f0 0x7fbf54153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.773+0000 7fbf58bb0640 1 -- 172.21.15.186:0/800921416 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf540c7ed0 msgr2=0x7fbf541f0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.773+0000 7fbf58bb0640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf540c7ed0 0x7fbf541f0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.773+0000 7fbf58bb0640 1 -- 172.21.15.186:0/800921416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf5406c7f0 msgr2=0x7fbf54153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.773+0000 7fbf58bb0640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf5406c7f0 0x7fbf54153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.773+0000 7fbf58bb0640 1 -- 172.21.15.186:0/800921416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf541efed0 con 0x7fbf5406cbd0 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.773+0000 7fbf58bb0640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf5406cbd0 0x7fbf541f03d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400e9d0 tx=0x7fbf4400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.773+0000 7fbf49ffb640 1 -- 172.21.15.186:0/800921416 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf44017070 con 0x7fbf5406cbd0 2024-03-28T15:01:24.775 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.773+0000 7fbf49ffb640 1 -- 172.21.15.186:0/800921416 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf4400ca00 con 0x7fbf5406cbd0 2024-03-28T15:01:24.776 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.773+0000 7fbf5b615640 1 -- 172.21.15.186:0/800921416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf5415b6e0 con 0x7fbf5406cbd0 2024-03-28T15:01:24.776 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.773+0000 7fbf49ffb640 1 -- 172.21.15.186:0/800921416 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4400cdb0 con 0x7fbf5406cbd0 2024-03-28T15:01:24.776 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.773+0000 7fbf5b615640 1 -- 172.21.15.186:0/800921416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf541f41d0 con 0x7fbf5406cbd0 2024-03-28T15:01:24.776 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.774+0000 7fbf5b615640 1 -- 172.21.15.186:0/800921416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf5404f8b0 con 0x7fbf5406cbd0 2024-03-28T15:01:24.776 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.775+0000 7fbf49ffb640 1 -- 172.21.15.186:0/800921416 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbf4401a070 con 0x7fbf5406cbd0 2024-03-28T15:01:24.777 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.775+0000 7fbf49ffb640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf3806c530 0x7fbf3806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:24.777 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.775+0000 7fbf49ffb640 1 -- 172.21.15.186:0/800921416 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fbf4408b1a0 con 0x7fbf5406cbd0 2024-03-28T15:01:24.777 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.775+0000 7fbf593b1640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf3806c530 0x7fbf3806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:24.777 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.776+0000 7fbf593b1640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf3806c530 0x7fbf3806e9f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbf50036000 tx=0x7fbf5003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:24.780 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.780+0000 7fbf49ffb640 1 -- 172.21.15.186:0/800921416 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fbf440594c0 con 0x7fbf5406cbd0 2024-03-28T15:01:24.956 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.955+0000 7fbf5b615640 1 -- 172.21.15.186:0/800921416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf5414e7f0 con 0x7fbf5406cbd0 2024-03-28T15:01:24.956 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.956+0000 7fbf49ffb640 1 -- 172.21.15.186:0/800921416 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12920 (secure 0 0 0) 0x7fbf4405d7c0 con 0x7fbf5406cbd0 2024-03-28T15:01:24.957 INFO:teuthology.misc.health.smithi186.stdout: 2024-03-28T15:01:24.957 INFO:teuthology.misc.health.smithi186.stdout:{"epoch":16,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:21.561039+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:24.959 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.958+0000 7fbf5b615640 1 -- 172.21.15.186:0/800921416 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf3806c530 msgr2=0x7fbf3806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:24.959 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.959+0000 7fbf5b615640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf3806c530 0x7fbf3806e9f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbf50036000 tx=0x7fbf5003b040 comp rx=0 tx=0).stop 2024-03-28T15:01:24.959 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.959+0000 7fbf5b615640 1 -- 172.21.15.186:0/800921416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf5406cbd0 msgr2=0x7fbf541f03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:24.959 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.959+0000 7fbf5b615640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf5406cbd0 0x7fbf541f03d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400e9d0 tx=0x7fbf4400ee90 comp rx=0 tx=0).stop 2024-03-28T15:01:24.959 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.959+0000 7fbf5b615640 1 -- 172.21.15.186:0/800921416 shutdown_connections 2024-03-28T15:01:24.959 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.959+0000 7fbf5b615640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf3806c530 0x7fbf3806e9f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:24.959 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.959+0000 7fbf5b615640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf540c7ed0 0x7fbf541f0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:24.959 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.959+0000 7fbf5b615640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf5406cbd0 0x7fbf541f03d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:24.960 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.959+0000 7fbf5b615640 1 --2- 172.21.15.186:0/800921416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf5406c7f0 0x7fbf54153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:24.960 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.959+0000 7fbf5b615640 1 -- 172.21.15.186:0/800921416 >> 172.21.15.186:0/800921416 conn(0x7fbf540ccb40 msgr2=0x7fbf5414fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:24.960 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.959+0000 7fbf5b615640 1 -- 172.21.15.186:0/800921416 shutdown_connections 2024-03-28T15:01:24.960 INFO:teuthology.misc.health.smithi186.stderr:2024-03-28T15:01:24.959+0000 7fbf5b615640 1 -- 172.21.15.186:0/800921416 wait complete. 2024-03-28T15:01:24.968 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-28T15:01:24.969 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-28T15:01:24.969 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T15:01:24.969 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T15:01:24.971 INFO:tasks.ceph.mds.a:Started 2024-03-28T15:01:24.971 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T15:01:24.972 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T15:01:24.973 INFO:tasks.ceph.mds.c:Started 2024-03-28T15:01:24.973 INFO:tasks.ceph.mds.e:Restarting daemon 2024-03-28T15:01:24.974 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-03-28T15:01:24.975 INFO:tasks.ceph.mds.e:Started 2024-03-28T15:01:24.975 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T15:01:24.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T15:01:24.978 INFO:tasks.ceph.mds.b:Started 2024-03-28T15:01:24.979 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T15:01:24.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T15:01:24.981 INFO:tasks.ceph.mds.d:Started 2024-03-28T15:01:24.981 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-28T15:01:24.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-03-28T15:01:24.983 INFO:tasks.ceph.mds.f:Started 2024-03-28T15:01:24.983 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-28T15:01:24.983 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:25.050 INFO:tasks.ceph.mds.a.smithi186.stdout:starting mds.a at 2024-03-28T15:01:25.050 INFO:tasks.ceph.mds.c.smithi186.stdout:starting mds.c at 2024-03-28T15:01:25.050 INFO:tasks.ceph.mds.e.smithi186.stdout:starting mds.e at 2024-03-28T15:01:25.066 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2024-03-28T15:01:25.067 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2024-03-28T15:01:25.067 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2024-03-28T15:01:25.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.092+0000 7f407a3c9640 1 Processor -- start 2024-03-28T15:01:25.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.092+0000 7f407a3c9640 1 -- start start 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.093+0000 7f407a3c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40740c2a50 0x7f40740c1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.093+0000 7f407a3c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40740c6b50 con 0x7f4074072680 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.093+0000 7f407a3c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4074072b90 con 0x7f4074157570 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.093+0000 7f407a3c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4074072e70 con 0x7f40740c2a50 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.094+0000 7f4078966640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4074072680 0x7f40740c6dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:52638/0 (socket says 172.21.15.186:52638) 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.094+0000 7f4078966640 1 -- 172.21.15.186:0/994822376 learned_addr learned my addr 172.21.15.186:0/994822376 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.094+0000 7f4073fff640 1 --2- 172.21.15.186:0/994822376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40740c2a50 0x7f40740c1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.094+0000 7f4072ffd640 1 -- 172.21.15.186:0/994822376 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1789165459 0 0) 0x7f40740c6b50 con 0x7f4074072680 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.094+0000 7f4072ffd640 1 -- 172.21.15.186:0/994822376 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f405c003520 con 0x7f4074072680 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.094+0000 7f4072ffd640 1 -- 172.21.15.186:0/994822376 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298385027 0 0) 0x7f4074072b90 con 0x7f4074157570 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.094+0000 7f4072ffd640 1 -- 172.21.15.186:0/994822376 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40740c6b50 con 0x7f4074157570 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.094+0000 7f4073fff640 1 -- 172.21.15.186:0/994822376 >> v1:172.21.15.191:6789/0 conn(0x7f4074157570 legacy=0x7f40740c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.094+0000 7f4073fff640 1 -- 172.21.15.186:0/994822376 >> v1:172.21.15.186:6789/0 conn(0x7f4074072680 legacy=0x7f40740c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.094+0000 7f4073fff640 1 -- 172.21.15.186:0/994822376 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4074156a90 con 0x7f40740c2a50 2024-03-28T15:01:25.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.095+0000 7f4073fff640 1 --2- 172.21.15.186:0/994822376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40740c2a50 0x7f40740c1830 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f40640087d0 tx=0x7f406402df50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f01590ba6f7501d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:25.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.095+0000 7f4072ffd640 1 -- 172.21.15.186:0/994822376 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4064034070 con 0x7f40740c2a50 2024-03-28T15:01:25.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.095+0000 7f4072ffd640 1 -- 172.21.15.186:0/994822376 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406402ed00 con 0x7f40740c2a50 2024-03-28T15:01:25.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.095+0000 7f4072ffd640 1 -- 172.21.15.186:0/994822376 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4064038600 con 0x7f40740c2a50 2024-03-28T15:01:25.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.095+0000 7f407a3c9640 1 -- 172.21.15.186:0/994822376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40740c2a50 msgr2=0x7f40740c1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.095+0000 7f407a3c9640 1 --2- 172.21.15.186:0/994822376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40740c2a50 0x7f40740c1830 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f40640087d0 tx=0x7f406402df50 comp rx=0 tx=0).stop 2024-03-28T15:01:25.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.095+0000 7f407a3c9640 1 -- 172.21.15.186:0/994822376 shutdown_connections 2024-03-28T15:01:25.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.095+0000 7f407a3c9640 1 --2- 172.21.15.186:0/994822376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40740c2a50 0x7f40740c1830 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.095+0000 7f407a3c9640 1 -- 172.21.15.186:0/994822376 >> 172.21.15.186:0/994822376 conn(0x7f40740c8a20 msgr2=0x7f40740c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:25.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.095+0000 7f407a3c9640 1 -- 172.21.15.186:0/994822376 shutdown_connections 2024-03-28T15:01:25.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.095+0000 7f407a3c9640 1 -- 172.21.15.186:0/994822376 wait complete. 2024-03-28T15:01:25.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.096+0000 7f407a3c9640 1 Processor -- start 2024-03-28T15:01:25.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.097+0000 7f407a3c9640 1 -- start start 2024-03-28T15:01:25.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.097+0000 7f407a3c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4074072680 0x7f40741ecc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.097+0000 7f407a3c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40740c2a50 0x7f40741ef570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.097+0000 7f407a3c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4074157570 0x7f40740c3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.097+0000 7f407a3c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f40740c1d70 con 0x7f4074072680 2024-03-28T15:01:25.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.097+0000 7f407a3c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4074157050 con 0x7f40740c2a50 2024-03-28T15:01:25.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.097+0000 7f407a3c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f40740c82a0 con 0x7f4074157570 2024-03-28T15:01:25.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4073fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4074072680 0x7f40741ecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4073fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4074072680 0x7f40741ecc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40164/0 (socket says 172.21.15.186:40164) 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4073fff640 1 -- 172.21.15.186:0/1113661299 learned_addr learned my addr 172.21.15.186:0/1113661299 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f40737fe640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40740c2a50 0x7f40741ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4078966640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4074157570 0x7f40740c3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4073fff640 1 -- 172.21.15.186:0/1113661299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4074157570 msgr2=0x7f40740c3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4073fff640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4074157570 0x7f40740c3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4073fff640 1 -- 172.21.15.186:0/1113661299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40740c2a50 msgr2=0x7f40741ef570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4073fff640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40740c2a50 0x7f40741ef570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4073fff640 1 -- 172.21.15.186:0/1113661299 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4074156200 con 0x7f4074072680 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f40737fe640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40740c2a50 0x7f40741ef570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4073fff640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4074072680 0x7f40741ecc00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f40640088f0 tx=0x7f40640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4070ff9640 1 -- 172.21.15.186:0/1113661299 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4064047030 con 0x7f4074072680 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4070ff9640 1 -- 172.21.15.186:0/1113661299 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406402eb00 con 0x7f4074072680 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f4070ff9640 1 -- 172.21.15.186:0/1113661299 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4064006e10 con 0x7f4074072680 2024-03-28T15:01:25.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f407a3c9640 1 -- 172.21.15.186:0/1113661299 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40741564e0 con 0x7f4074072680 2024-03-28T15:01:25.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.098+0000 7f407a3c9640 1 -- 172.21.15.186:0/1113661299 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40741ec400 con 0x7f4074072680 2024-03-28T15:01:25.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.099+0000 7f407a3c9640 1 -- 172.21.15.186:0/1113661299 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4074069500 con 0x7f4074072680 2024-03-28T15:01:25.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.100+0000 7f4070ff9640 1 -- 172.21.15.186:0/1113661299 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f406404a070 con 0x7f4074072680 2024-03-28T15:01:25.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.100+0000 7f4070ff9640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f406006c560 0x7f406006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.100+0000 7f4070ff9640 1 -- 172.21.15.186:0/1113661299 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f40640af990 con 0x7f4074072680 2024-03-28T15:01:25.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.100+0000 7f40737fe640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f406006c560 0x7f406006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.100+0000 7f40737fe640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f406006c560 0x7f406006ea20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f40740b17d0 tx=0x7f4068009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:25.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.104+0000 7f4070ff9640 1 -- 172.21.15.186:0/1113661299 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4064051070 con 0x7f4074072680 2024-03-28T15:01:25.171 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:01:25.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:01:25.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.286+0000 7f407a3c9640 1 -- 172.21.15.186:0/1113661299 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4074071650 con 0x7f4074072680 2024-03-28T15:01:25.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.287+0000 7f4070ff9640 1 -- 172.21.15.186:0/1113661299 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12920 (secure 0 0 0) 0x7f406402ecd0 con 0x7f4074072680 2024-03-28T15:01:25.287 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:25.287 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":16,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:21.561039+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:25.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.289+0000 7f407a3c9640 1 -- 172.21.15.186:0/1113661299 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f406006c560 msgr2=0x7f406006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.289+0000 7f407a3c9640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f406006c560 0x7f406006ea20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f40740b17d0 tx=0x7f4068009040 comp rx=0 tx=0).stop 2024-03-28T15:01:25.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.289+0000 7f407a3c9640 1 -- 172.21.15.186:0/1113661299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4074072680 msgr2=0x7f40741ecc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.289+0000 7f407a3c9640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4074072680 0x7f40741ecc00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f40640088f0 tx=0x7f40640046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.289+0000 7f407a3c9640 1 -- 172.21.15.186:0/1113661299 shutdown_connections 2024-03-28T15:01:25.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.289+0000 7f407a3c9640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f406006c560 0x7f406006ea20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.289+0000 7f407a3c9640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4074157570 0x7f40740c3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.289+0000 7f407a3c9640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40740c2a50 0x7f40741ef570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.289+0000 7f407a3c9640 1 --2- 172.21.15.186:0/1113661299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4074072680 0x7f40741ecc00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.289+0000 7f407a3c9640 1 -- 172.21.15.186:0/1113661299 >> 172.21.15.186:0/1113661299 conn(0x7f40740c8a20 msgr2=0x7f40740c1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:25.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.290+0000 7f407a3c9640 1 -- 172.21.15.186:0/1113661299 shutdown_connections 2024-03-28T15:01:25.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.290+0000 7f407a3c9640 1 -- 172.21.15.186:0/1113661299 wait complete. 2024-03-28T15:01:25.301 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:01:18.661274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:01:25.301 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:01:25.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.405+0000 7f2ac3ddd640 1 Processor -- start 2024-03-28T15:01:25.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.405+0000 7f2ac3ddd640 1 -- start start 2024-03-28T15:01:25.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac3ddd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc0701e0 0x7f2abc0c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac3ddd640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2abc15ed90 con 0x7f2abc154c40 2024-03-28T15:01:25.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac3ddd640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2abc15a860 con 0x7f2abc0c78b0 2024-03-28T15:01:25.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac3ddd640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f2abc155020 con 0x7f2abc0701e0 2024-03-28T15:01:25.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac1b79640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f2abc154c40 0x7f2abc0cc5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:52654/0 (socket says 172.21.15.186:52654) 2024-03-28T15:01:25.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac1b79640 1 -- 172.21.15.186:0/2972003142 learned_addr learned my addr 172.21.15.186:0/2972003142 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac1378640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc0701e0 0x7f2abc0c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2984122687 0 0) 0x7f2abc15ed90 con 0x7f2abc154c40 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa8003520 con 0x7f2abc154c40 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2889480963 0 0) 0x7f2abc15a860 con 0x7f2abc0c78b0 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2abc15ed90 con 0x7f2abc0c78b0 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4044533055 0 0) 0x7f2aa8003520 con 0x7f2abc154c40 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc0701e0 msgr2=0x7f2abc0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc0701e0 0x7f2abc0c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 >> v1:172.21.15.191:6789/0 conn(0x7f2abc0c78b0 legacy=0x7f2abc15ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc15eab0 con 0x7f2abc154c40 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f2ab8001670 con 0x7f2abc154c40 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 >> v1:172.21.15.186:6789/0 conn(0x7f2abc154c40 legacy=0x7f2abc0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc0701e0 0x7f2aa8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 0x7f2aa8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2aa8004d40 0x7f2aa80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc155020 con 0x7f2aa8004120 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa800bb80 con 0x7f2aa8004d40 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.406+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f2aa800bd80 con 0x7f2abc0701e0 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.407+0000 7f2ac1378640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 0x7f2aa8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.407+0000 7f2ac1b79640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc0701e0 0x7f2aa8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.407+0000 7f2ac237a640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2aa8004d40 0x7f2aa80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.407+0000 7f2ac1378640 1 -- 172.21.15.186:0/2972003142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc0701e0 msgr2=0x7f2aa8003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.407+0000 7f2ac1378640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc0701e0 0x7f2aa8003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.407+0000 7f2ac1378640 1 -- 172.21.15.186:0/2972003142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2aa8004d40 msgr2=0x7f2aa80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.407+0000 7f2ac1378640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2aa8004d40 0x7f2aa80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.407+0000 7f2ac1378640 1 -- 172.21.15.186:0/2972003142 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa8003520 con 0x7f2aa8004120 2024-03-28T15:01:25.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.407+0000 7f2ac1378640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 0x7f2aa8004800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009ac0 tx=0x7f2aa8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701d84b00370261f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.408+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac039070 con 0x7f2aa8004120 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.408+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aac02c6b0 con 0x7f2aa8004120 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.408+0000 7f2ac0b77640 1 -- 172.21.15.186:0/2972003142 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac009690 con 0x7f2aa8004120 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.408+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/2972003142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 msgr2=0x7f2aa8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.408+0000 7f2ac3ddd640 1 --2- 172.21.15.186:0/2972003142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 0x7f2aa8004800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009ac0 tx=0x7f2aa8005830 comp rx=0 tx=0).stop 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.408+0000 7f2ac237a640 1 -- 172.21.15.186:0/2972003142 reap_dead start 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.408+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/2972003142 shutdown_connections 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.408+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/2972003142 >> 172.21.15.186:0/2972003142 conn(0x7f2abc06df10 msgr2=0x7f2abc06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.408+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/2972003142 shutdown_connections 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.408+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/2972003142 wait complete. 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.409+0000 7f2ac3ddd640 1 Processor -- start 2024-03-28T15:01:25.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.409+0000 7f2ac3ddd640 1 -- start start 2024-03-28T15:01:25.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac3ddd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 0x7f2abc0c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac3ddd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc155b70 0x7f2abc159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac3ddd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2abc159e50 0x7f2abc14de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac3ddd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc0c7470 con 0x7f2aa8004120 2024-03-28T15:01:25.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac3ddd640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc15e860 con 0x7f2abc159e50 2024-03-28T15:01:25.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac3ddd640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f2abc0ccd00 con 0x7f2abc155b70 2024-03-28T15:01:25.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac1b79640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 0x7f2abc0c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac1b79640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 0x7f2abc0c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40172/0 (socket says 172.21.15.186:40172) 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac1b79640 1 -- 172.21.15.186:0/877773466 learned_addr learned my addr 172.21.15.186:0/877773466 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac237a640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2abc159e50 0x7f2abc14de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac1378640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc155b70 0x7f2abc159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac1b79640 1 -- 172.21.15.186:0/877773466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc155b70 msgr2=0x7f2abc159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac1b79640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc155b70 0x7f2abc159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac1b79640 1 -- 172.21.15.186:0/877773466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2abc159e50 msgr2=0x7f2abc14de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac1b79640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2abc159e50 0x7f2abc14de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac1b79640 1 -- 172.21.15.186:0/877773466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc1566d0 con 0x7f2aa8004120 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac1378640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc155b70 0x7f2abc159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.410+0000 7f2ac1b79640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 0x7f2abc0c5500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8004e30 tx=0x7f2ab8007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.411+0000 7f2ab27fc640 1 -- 172.21.15.186:0/877773466 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800f7b0 con 0x7f2aa8004120 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.411+0000 7f2ab27fc640 1 -- 172.21.15.186:0/877773466 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab8018070 con 0x7f2aa8004120 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.411+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/877773466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2abc15a7d0 con 0x7f2aa8004120 2024-03-28T15:01:25.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.411+0000 7f2ab27fc640 1 -- 172.21.15.186:0/877773466 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800fbb0 con 0x7f2aa8004120 2024-03-28T15:01:25.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.411+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/877773466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2abc158b50 con 0x7f2aa8004120 2024-03-28T15:01:25.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.412+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/877773466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2abc069570 con 0x7f2aa8004120 2024-03-28T15:01:25.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.412+0000 7f2ab27fc640 1 -- 172.21.15.186:0/877773466 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2ab800fd50 con 0x7f2aa8004120 2024-03-28T15:01:25.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.412+0000 7f2ab27fc640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2aa006c560 0x7f2aa006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.413+0000 7f2ab27fc640 1 -- 172.21.15.186:0/877773466 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f2ab808b240 con 0x7f2aa8004120 2024-03-28T15:01:25.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.413+0000 7f2ac1378640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2aa006c560 0x7f2aa006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.413+0000 7f2ac1378640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2aa006c560 0x7f2aa006ea20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2aac036000 tx=0x7f2aac030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:25.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.417+0000 7f2ab27fc640 1 -- 172.21.15.186:0/877773466 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2ab8059bb0 con 0x7f2aa8004120 2024-03-28T15:01:25.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.592+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/877773466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2abc153a20 con 0x7f2aa8004120 2024-03-28T15:01:25.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.592+0000 7f2ab27fc640 1 -- 172.21.15.186:0/877773466 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f2ab80594b0 con 0x7f2aa8004120 2024-03-28T15:01:25.593 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 1 2024-03-28T15:01:25.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/877773466 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2aa006c560 msgr2=0x7f2aa006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2aa006c560 0x7f2aa006ea20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2aac036000 tx=0x7f2aac030000 comp rx=0 tx=0).stop 2024-03-28T15:01:25.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/877773466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 msgr2=0x7f2abc0c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 0x7f2abc0c5500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8004e30 tx=0x7f2ab8007430 comp rx=0 tx=0).stop 2024-03-28T15:01:25.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/877773466 shutdown_connections 2024-03-28T15:01:25.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2aa006c560 0x7f2aa006ea20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2abc159e50 0x7f2abc14de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc155b70 0x7f2abc159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 --2- 172.21.15.186:0/877773466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2aa8004120 0x7f2abc0c5500 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/877773466 >> 172.21.15.186:0/877773466 conn(0x7f2abc06df10 msgr2=0x7f2abc152560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:25.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/877773466 shutdown_connections 2024-03-28T15:01:25.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.594+0000 7f2ac3ddd640 1 -- 172.21.15.186:0/877773466 wait complete. 2024-03-28T15:01:25.605 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:01:25.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.706+0000 7f58e835e640 1 Processor -- start 2024-03-28T15:01:25.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.706+0000 7f58e835e640 1 -- start start 2024-03-28T15:01:25.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.707+0000 7f58e835e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e00c5380 0x7f58e00cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.707+0000 7f58e835e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58e00c6f90 con 0x7f58e00caa70 2024-03-28T15:01:25.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.707+0000 7f58e835e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58e00cc970 con 0x7f58e00cb390 2024-03-28T15:01:25.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.707+0000 7f58e835e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f58e0071a80 con 0x7f58e00c5380 2024-03-28T15:01:25.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.707+0000 7f58e60fa640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f58e00caa70 0x7f58e00c64c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:52668/0 (socket says 172.21.15.186:52668) 2024-03-28T15:01:25.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.707+0000 7f58e60fa640 1 -- 172.21.15.186:0/3911683666 learned_addr learned my addr 172.21.15.186:0/3911683666 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:25.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e58f9640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e00c5380 0x7f58e00cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2921571087 0 0) 0x7f58dc001490 con 0x7f58e00caa70 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58cc003520 con 0x7f58e00caa70 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 260067546 0 0) 0x7f58cc003520 con 0x7f58e00caa70 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e00c5380 msgr2=0x7f58e00cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e00c5380 0x7f58e00cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 >> v1:172.21.15.191:6789/0 conn(0x7f58e00cb390 legacy=0x7f58e0165220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e00cc690 con 0x7f58e00caa70 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f58dc001bc0 con 0x7f58e00caa70 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 >> v1:172.21.15.186:6789/0 conn(0x7f58e00caa70 legacy=0x7f58e00c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e00c5380 0x7f58cc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58cc003f20 0x7f58cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 0x7f58cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc00b9f0 con 0x7f58cc004d40 2024-03-28T15:01:25.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc00bba0 con 0x7f58cc003f20 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f58cc00bda0 con 0x7f58e00c5380 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e60fa640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e00c5380 0x7f58cc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e68fb640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 0x7f58cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e58f9640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58cc003f20 0x7f58cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e68fb640 1 -- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e00c5380 msgr2=0x7f58cc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e68fb640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e00c5380 0x7f58cc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e68fb640 1 -- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58cc003f20 msgr2=0x7f58cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e68fb640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58cc003f20 0x7f58cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e68fb640 1 -- 172.21.15.186:0/3911683666 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58cc003520 con 0x7f58cc004d40 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.708+0000 7f58e60fa640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e00c5380 0x7f58cc0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.709+0000 7f58e68fb640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 0x7f58cc0051c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f58e00ccb40 tx=0x7f58cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de2bc3dc9ded64a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.709+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d802f050 con 0x7f58cc004d40 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.709+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d80052f0 con 0x7f58cc004d40 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.709+0000 7f58e50f8640 1 -- 172.21.15.186:0/3911683666 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d8001520 con 0x7f58cc004d40 2024-03-28T15:01:25.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.710+0000 7f58e835e640 1 -- 172.21.15.186:0/3911683666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 msgr2=0x7f58cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.710+0000 7f58e835e640 1 --2- 172.21.15.186:0/3911683666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 0x7f58cc0051c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f58e00ccb40 tx=0x7f58cc005830 comp rx=0 tx=0).stop 2024-03-28T15:01:25.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.710+0000 7f58e68fb640 1 -- 172.21.15.186:0/3911683666 reap_dead start 2024-03-28T15:01:25.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.710+0000 7f58e835e640 1 -- 172.21.15.186:0/3911683666 shutdown_connections 2024-03-28T15:01:25.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.710+0000 7f58e835e640 1 -- 172.21.15.186:0/3911683666 >> 172.21.15.186:0/3911683666 conn(0x7f58e00d00f0 msgr2=0x7f58e00d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:25.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.710+0000 7f58e835e640 1 -- 172.21.15.186:0/3911683666 shutdown_connections 2024-03-28T15:01:25.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.710+0000 7f58e835e640 1 -- 172.21.15.186:0/3911683666 wait complete. 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.711+0000 7f58e835e640 1 Processor -- start 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.711+0000 7f58e835e640 1 -- start start 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e835e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 0x7f58e00b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e835e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58e00c86b0 0x7f58e01e6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e835e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e01e6890 0x7f58e00c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e835e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f58e00c5cc0 con 0x7f58cc004d40 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e835e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f58e00c6030 con 0x7f58e00c86b0 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e835e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f58e015b810 con 0x7f58e01e6890 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e60fa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 0x7f58e00b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e60fa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 0x7f58e00b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40202/0 (socket says 172.21.15.186:40202) 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e60fa640 1 -- 172.21.15.186:0/3138161240 learned_addr learned my addr 172.21.15.186:0/3138161240 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e68fb640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e01e6890 0x7f58e00c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e58f9640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58e00c86b0 0x7f58e01e6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e60fa640 1 -- 172.21.15.186:0/3138161240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e01e6890 msgr2=0x7f58e00c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e60fa640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e01e6890 0x7f58e00c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e60fa640 1 -- 172.21.15.186:0/3138161240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58e00c86b0 msgr2=0x7f58e01e6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e60fa640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58e00c86b0 0x7f58e01e6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e60fa640 1 -- 172.21.15.186:0/3138161240 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e0159f50 con 0x7f58cc004d40 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e58f9640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58e00c86b0 0x7f58e01e6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e68fb640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e01e6890 0x7f58e00c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e60fa640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 0x7f58e00b5ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f58dc00add0 tx=0x7f58dc007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58d6ffd640 1 -- 172.21.15.186:0/3138161240 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc017020 con 0x7f58cc004d40 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58d6ffd640 1 -- 172.21.15.186:0/3138161240 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58dc013600 con 0x7f58cc004d40 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e835e640 1 -- 172.21.15.186:0/3138161240 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58e015a1f0 con 0x7f58cc004d40 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58d6ffd640 1 -- 172.21.15.186:0/3138161240 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc0139d0 con 0x7f58cc004d40 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.712+0000 7f58e835e640 1 -- 172.21.15.186:0/3138161240 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58e01f07f0 con 0x7f58cc004d40 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.713+0000 7f58e835e640 1 -- 172.21.15.186:0/3138161240 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58e0072fe0 con 0x7f58cc004d40 2024-03-28T15:01:25.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.714+0000 7f58d6ffd640 1 -- 172.21.15.186:0/3138161240 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f58dc01a020 con 0x7f58cc004d40 2024-03-28T15:01:25.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.714+0000 7f58d6ffd640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f58c406c560 0x7f58c406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:25.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.714+0000 7f58d6ffd640 1 -- 172.21.15.186:0/3138161240 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f58dc08c030 con 0x7f58cc004d40 2024-03-28T15:01:25.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.714+0000 7f58e58f9640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f58c406c560 0x7f58c406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:25.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.714+0000 7f58e58f9640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f58c406c560 0x7f58c406ea20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f58e00c77b0 tx=0x7f58d0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:25.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.719+0000 7f58d6ffd640 1 -- 172.21.15.186:0/3138161240 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f58e0072fe0 con 0x7f58cc004d40 2024-03-28T15:01:25.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.930+0000 7f58e835e640 1 -- 172.21.15.186:0/3138161240 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58e00686d0 con 0x7f58cc004d40 2024-03-28T15:01:25.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.931+0000 7f58d6ffd640 1 -- 172.21.15.186:0/3138161240 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1395 (secure 0 0 0) 0x7f58dc021020 con 0x7f58cc004d40 2024-03-28T15:01:25.931 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:25.931 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-28T15:01:25.931 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 2 2024-03-28T15:01:25.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 -- 172.21.15.186:0/3138161240 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f58c406c560 msgr2=0x7f58c406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f58c406c560 0x7f58c406ea20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f58e00c77b0 tx=0x7f58d0008040 comp rx=0 tx=0).stop 2024-03-28T15:01:25.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 -- 172.21.15.186:0/3138161240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 msgr2=0x7f58e00b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:25.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 0x7f58e00b5ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f58dc00add0 tx=0x7f58dc007430 comp rx=0 tx=0).stop 2024-03-28T15:01:25.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 -- 172.21.15.186:0/3138161240 shutdown_connections 2024-03-28T15:01:25.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f58c406c560 0x7f58c406ea20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58e01e6890 0x7f58e00c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58e00c86b0 0x7f58e01e6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 --2- 172.21.15.186:0/3138161240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58cc004d40 0x7f58e00b5ae0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:25.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 -- 172.21.15.186:0/3138161240 >> 172.21.15.186:0/3138161240 conn(0x7f58e00d00f0 msgr2=0x7f58e00cf330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:25.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 -- 172.21.15.186:0/3138161240 shutdown_connections 2024-03-28T15:01:25.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:25.933+0000 7f58e835e640 1 -- 172.21.15.186:0/3138161240 wait complete. 2024-03-28T15:01:27.944 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:01:28.057 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.057+0000 7f5bd7081640 1 Processor -- start 2024-03-28T15:01:28.057 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.057+0000 7f5bd7081640 1 -- start start 2024-03-28T15:01:28.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd7081640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5bd00c4700 0x7f5bd00c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd7081640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5bd0159460 0x7f5bd0152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd7081640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bd006fd00 con 0x7f5bd01527b0 2024-03-28T15:01:28.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd7081640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f5bd0062f00 con 0x7f5bd00c4700 2024-03-28T15:01:28.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd7081640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f5bd006ffe0 con 0x7f5bd0159460 2024-03-28T15:01:28.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd4e1d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5bd00c4700 0x7f5bd00c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd4e1d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5bd00c4700 0x7f5bd00c4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40218/0 (socket says 172.21.15.186:40218) 2024-03-28T15:01:28.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd4e1d640 1 -- 172.21.15.186:0/1868821030 learned_addr learned my addr 172.21.15.186:0/1868821030 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:28.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd561e640 1 --2- 172.21.15.186:0/1868821030 >> v2:172.21.15.191:3300/0 conn(0x7f5bd0159460 0x7f5bd0152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd4e1d640 1 -- 172.21.15.186:0/1868821030 >> v2:172.21.15.191:3300/0 conn(0x7f5bd0159460 msgr2=0x7f5bd0152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:28.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd4e1d640 1 --2- 172.21.15.186:0/1868821030 >> v2:172.21.15.191:3300/0 conn(0x7f5bd0159460 0x7f5bd0152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd4e1d640 1 -- 172.21.15.186:0/1868821030 >> v1:172.21.15.191:6789/0 conn(0x7f5bd01527b0 legacy=0x7f5bd0158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.058+0000 7f5bd4e1d640 1 -- 172.21.15.186:0/1868821030 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd0062940 con 0x7f5bd00c4700 2024-03-28T15:01:28.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.059+0000 7f5bd4e1d640 1 --2- 172.21.15.186:0/1868821030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd00c4700 0x7f5bd00c4ae0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00c65d0 tx=0x7f5bc402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c2be65c5b9f7fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:28.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.059+0000 7f5bcf7fe640 1 -- 172.21.15.186:0/1868821030 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc402f850 con 0x7f5bd00c4700 2024-03-28T15:01:28.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.059+0000 7f5bcf7fe640 1 -- 172.21.15.186:0/1868821030 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc402f9f0 con 0x7f5bd00c4700 2024-03-28T15:01:28.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.059+0000 7f5bcf7fe640 1 -- 172.21.15.186:0/1868821030 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc402fdc0 con 0x7f5bd00c4700 2024-03-28T15:01:28.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.060+0000 7f5bd7081640 1 -- 172.21.15.186:0/1868821030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd00c4700 msgr2=0x7f5bd00c4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.060+0000 7f5bd7081640 1 --2- 172.21.15.186:0/1868821030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd00c4700 0x7f5bd00c4ae0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00c65d0 tx=0x7f5bc402ba30 comp rx=0 tx=0).stop 2024-03-28T15:01:28.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.060+0000 7f5bd7081640 1 -- 172.21.15.186:0/1868821030 shutdown_connections 2024-03-28T15:01:28.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.060+0000 7f5bd7081640 1 --2- 172.21.15.186:0/1868821030 >> v2:172.21.15.191:3300/0 conn(0x7f5bd0159460 0x7f5bd0152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.060+0000 7f5bd7081640 1 --2- 172.21.15.186:0/1868821030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd00c4700 0x7f5bd00c4ae0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.060+0000 7f5bd7081640 1 -- 172.21.15.186:0/1868821030 >> 172.21.15.186:0/1868821030 conn(0x7f5bd00cc580 msgr2=0x7f5bd00cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:28.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.060+0000 7f5bd7081640 1 -- 172.21.15.186:0/1868821030 shutdown_connections 2024-03-28T15:01:28.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.060+0000 7f5bd7081640 1 -- 172.21.15.186:0/1868821030 wait complete. 2024-03-28T15:01:28.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.061+0000 7f5bd7081640 1 Processor -- start 2024-03-28T15:01:28.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.061+0000 7f5bd7081640 1 -- start start 2024-03-28T15:01:28.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd7081640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd00c4700 0x7f5bd01f1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd7081640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bd01527b0 0x7f5bd01f4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd7081640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bd0159460 0x7f5bd01f1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd7081640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd00c66b0 con 0x7f5bd00c4700 2024-03-28T15:01:28.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd7081640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd00c8df0 con 0x7f5bd0159460 2024-03-28T15:01:28.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd7081640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5bd00c6830 con 0x7f5bd01527b0 2024-03-28T15:01:28.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd4e1d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd00c4700 0x7f5bd01f1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd4e1d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd00c4700 0x7f5bd01f1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40228/0 (socket says 172.21.15.186:40228) 2024-03-28T15:01:28.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd4e1d640 1 -- 172.21.15.186:0/1236863569 learned_addr learned my addr 172.21.15.186:0/1236863569 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:28.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bcffff640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bd01527b0 0x7f5bd01f4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd561e640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bd0159460 0x7f5bd01f1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd4e1d640 1 -- 172.21.15.186:0/1236863569 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bd01527b0 msgr2=0x7f5bd01f4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd4e1d640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bd01527b0 0x7f5bd01f4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd4e1d640 1 -- 172.21.15.186:0/1236863569 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bd0159460 msgr2=0x7f5bd01f1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd4e1d640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bd0159460 0x7f5bd01f1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.062+0000 7f5bd4e1d640 1 -- 172.21.15.186:0/1236863569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd00cc330 con 0x7f5bd00c4700 2024-03-28T15:01:28.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.063+0000 7f5bd4e1d640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd00c4700 0x7f5bd01f1310 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5bc402ba70 tx=0x7f5bc4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:28.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.066+0000 7f5bcd7fa640 1 -- 172.21.15.186:0/1236863569 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4037c90 con 0x7f5bd00c4700 2024-03-28T15:01:28.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.066+0000 7f5bcd7fa640 1 -- 172.21.15.186:0/1236863569 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc4046070 con 0x7f5bd00c4700 2024-03-28T15:01:28.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.066+0000 7f5bd7081640 1 -- 172.21.15.186:0/1236863569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd0157dd0 con 0x7f5bd00c4700 2024-03-28T15:01:28.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.066+0000 7f5bcd7fa640 1 -- 172.21.15.186:0/1236863569 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4041640 con 0x7f5bd00c4700 2024-03-28T15:01:28.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.066+0000 7f5bd7081640 1 -- 172.21.15.186:0/1236863569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd0158090 con 0x7f5bd00c4700 2024-03-28T15:01:28.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.067+0000 7f5bd7081640 1 -- 172.21.15.186:0/1236863569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bd0069500 con 0x7f5bd00c4700 2024-03-28T15:01:28.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.067+0000 7f5bcd7fa640 1 -- 172.21.15.186:0/1236863569 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5bc4040070 con 0x7f5bd00c4700 2024-03-28T15:01:28.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.067+0000 7f5bcd7fa640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bb406c530 0x7f5bb406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.068+0000 7f5bcd7fa640 1 -- 172.21.15.186:0/1236863569 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f5bc40adaa0 con 0x7f5bd00c4700 2024-03-28T15:01:28.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.068+0000 7f5bcffff640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bb406c530 0x7f5bb406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.068+0000 7f5bcffff640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bb406c530 0x7f5bb406e9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0157ca0 tx=0x7f5bbc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:28.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.072+0000 7f5bcd7fa640 1 -- 172.21.15.186:0/1236863569 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5bc4033030 con 0x7f5bd00c4700 2024-03-28T15:01:28.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.288+0000 7f5bd7081640 1 -- 172.21.15.186:0/1236863569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bd0157860 con 0x7f5bd00c4700 2024-03-28T15:01:28.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.289+0000 7f5bcd7fa640 1 -- 172.21.15.186:0/1236863569 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5527 (secure 0 0 0) 0x7f5bc407bd30 con 0x7f5bd00c4700 2024-03-28T15:01:28.289 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:28.289 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4308,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-28T15:01:28.289 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 3 2024-03-28T15:01:28.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 -- 172.21.15.186:0/1236863569 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bb406c530 msgr2=0x7f5bb406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bb406c530 0x7f5bb406e9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0157ca0 tx=0x7f5bbc009040 comp rx=0 tx=0).stop 2024-03-28T15:01:28.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 -- 172.21.15.186:0/1236863569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd00c4700 msgr2=0x7f5bd01f1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd00c4700 0x7f5bd01f1310 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5bc402ba70 tx=0x7f5bc4004730 comp rx=0 tx=0).stop 2024-03-28T15:01:28.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 -- 172.21.15.186:0/1236863569 shutdown_connections 2024-03-28T15:01:28.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bb406c530 0x7f5bb406e9f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bd0159460 0x7f5bd01f1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bd01527b0 0x7f5bd01f4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 --2- 172.21.15.186:0/1236863569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd00c4700 0x7f5bd01f1310 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 -- 172.21.15.186:0/1236863569 >> 172.21.15.186:0/1236863569 conn(0x7f5bd00cc580 msgr2=0x7f5bd01510b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:28.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 -- 172.21.15.186:0/1236863569 shutdown_connections 2024-03-28T15:01:28.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.291+0000 7f5bd7081640 1 -- 172.21.15.186:0/1236863569 wait complete. 2024-03-28T15:01:28.302 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:28.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.402+0000 7fbdcb57a640 1 Processor -- start 2024-03-28T15:01:28.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.402+0000 7fbdcb57a640 1 -- start start 2024-03-28T15:01:28.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdcb57a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbdc4071b40 0x7fbdc415ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdcb57a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc40c8350 0x7fbdc415b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdcb57a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc415b8d0 con 0x7fbdc4154f00 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdcb57a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fbdc406f170 con 0x7fbdc4071b40 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdcb57a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fbdc406f450 con 0x7fbdc40c8350 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdc9316640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fbdc4154f00 0x7fbdc406f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:52684/0 (socket says 172.21.15.186:52684) 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdc9316640 1 -- 172.21.15.186:0/1048958713 learned_addr learned my addr 172.21.15.186:0/1048958713 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1805601864 0 0) 0x7fbdc415b8d0 con 0x7fbdc4154f00 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdb0003520 con 0x7fbdc4154f00 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdc8b15640 1 --2- 172.21.15.186:0/1048958713 >> v2:172.21.15.191:3300/0 conn(0x7fbdc4071b40 0x7fbdc415ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdc9b17640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc40c8350 0x7fbdc415b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2621897928 0 0) 0x7fbdb0003520 con 0x7fbdc4154f00 2024-03-28T15:01:28.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc40c8350 msgr2=0x7fbdc415b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:28.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc40c8350 0x7fbdc415b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 >> v2:172.21.15.191:3300/0 conn(0x7fbdc4071b40 msgr2=0x7fbdc415ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:28.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 --2- 172.21.15.186:0/1048958713 >> v2:172.21.15.191:3300/0 conn(0x7fbdc4071b40 0x7fbdc415ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc40c7b10 con 0x7fbdc4154f00 2024-03-28T15:01:28.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7fbdc0001670 con 0x7fbdc4154f00 2024-03-28T15:01:28.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 >> v1:172.21.15.186:6789/0 conn(0x7fbdc4154f00 legacy=0x7fbdc406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdb00039b0 0x7fbdb0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc40c8350 0x7fbdb0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 0x7fbdb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc406f170 con 0x7fbdb0004d40 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc406f450 con 0x7fbdb00039b0 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fbdb000baf0 con 0x7fbdc40c8350 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdc9b17640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 0x7fbdb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdc9316640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdb00039b0 0x7fbdb0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.403+0000 7fbdc8b15640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc40c8350 0x7fbdb0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.404+0000 7fbdc9b17640 1 -- 172.21.15.186:0/1048958713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc40c8350 msgr2=0x7fbdb0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.404+0000 7fbdc9b17640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc40c8350 0x7fbdb0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.404+0000 7fbdc9b17640 1 -- 172.21.15.186:0/1048958713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdb00039b0 msgr2=0x7fbdb0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.404+0000 7fbdc9b17640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdb00039b0 0x7fbdb0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.404+0000 7fbdc9b17640 1 -- 172.21.15.186:0/1048958713 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb0003520 con 0x7fbdb0004d40 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.404+0000 7fbdc9b17640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 0x7fbdb00051c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc007330 tx=0x7fbdb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3a18ec198d9e0a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.404+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc039070 con 0x7fbdb0004d40 2024-03-28T15:01:28.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.404+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdbc02cca0 con 0x7fbdb0004d40 2024-03-28T15:01:28.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.404+0000 7fbdb7fff640 1 -- 172.21.15.186:0/1048958713 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc009be0 con 0x7fbdb0004d40 2024-03-28T15:01:28.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.405+0000 7fbdcb57a640 1 -- 172.21.15.186:0/1048958713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 msgr2=0x7fbdb00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.405+0000 7fbdcb57a640 1 --2- 172.21.15.186:0/1048958713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 0x7fbdb00051c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc007330 tx=0x7fbdb0005830 comp rx=0 tx=0).stop 2024-03-28T15:01:28.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.405+0000 7fbdc9b17640 1 -- 172.21.15.186:0/1048958713 reap_dead start 2024-03-28T15:01:28.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.405+0000 7fbdcb57a640 1 -- 172.21.15.186:0/1048958713 shutdown_connections 2024-03-28T15:01:28.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.405+0000 7fbdcb57a640 1 -- 172.21.15.186:0/1048958713 >> 172.21.15.186:0/1048958713 conn(0x7fbdc40cc580 msgr2=0x7fbdc40cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:28.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.405+0000 7fbdcb57a640 1 -- 172.21.15.186:0/1048958713 shutdown_connections 2024-03-28T15:01:28.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.405+0000 7fbdcb57a640 1 -- 172.21.15.186:0/1048958713 wait complete. 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.406+0000 7fbdcb57a640 1 Processor -- start 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.406+0000 7fbdcb57a640 1 -- start start 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdcb57a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 0x7fbdc4153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdcb57a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc414c630 0x7fbdc4159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdcb57a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdc40c59e0 0x7fbdc40c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdcb57a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc406fff0 con 0x7fbdb0004d40 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdcb57a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc41552e0 con 0x7fbdc40c59e0 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdcb57a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fbdc4070170 con 0x7fbdc414c630 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc9316640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 0x7fbdc4153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc9316640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 0x7fbdc4153900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40238/0 (socket says 172.21.15.186:40238) 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc9316640 1 -- 172.21.15.186:0/3757235604 learned_addr learned my addr 172.21.15.186:0/3757235604 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:28.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc8b15640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc414c630 0x7fbdc4159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc9b17640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdc40c59e0 0x7fbdc40c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc9316640 1 -- 172.21.15.186:0/3757235604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc414c630 msgr2=0x7fbdc4159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc9316640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc414c630 0x7fbdc4159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc9316640 1 -- 172.21.15.186:0/3757235604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdc40c59e0 msgr2=0x7fbdc40c5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc9316640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdc40c59e0 0x7fbdc40c5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc9316640 1 -- 172.21.15.186:0/3757235604 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc40c63a0 con 0x7fbdb0004d40 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc9b17640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdc40c59e0 0x7fbdc40c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.407+0000 7fbdc9316640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 0x7fbdc4153900 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0007340 tx=0x7fbdc0007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.408+0000 7fbdb5ffb640 1 -- 172.21.15.186:0/3757235604 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc001a070 con 0x7fbdb0004d40 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.408+0000 7fbdb5ffb640 1 -- 172.21.15.186:0/3757235604 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc000f940 con 0x7fbdb0004d40 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.408+0000 7fbdcb57a640 1 -- 172.21.15.186:0/3757235604 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc41f17c0 con 0x7fbdb0004d40 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.408+0000 7fbdb5ffb640 1 -- 172.21.15.186:0/3757235604 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc000fd10 con 0x7fbdb0004d40 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.408+0000 7fbdcb57a640 1 -- 172.21.15.186:0/3757235604 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc41f0f20 con 0x7fbdb0004d40 2024-03-28T15:01:28.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.409+0000 7fbdcb57a640 1 -- 172.21.15.186:0/3757235604 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdc4150a10 con 0x7fbdb0004d40 2024-03-28T15:01:28.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.409+0000 7fbdb5ffb640 1 -- 172.21.15.186:0/3757235604 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbdc001d070 con 0x7fbdb0004d40 2024-03-28T15:01:28.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.409+0000 7fbdb5ffb640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbda806c530 0x7fbda806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.409+0000 7fbdb5ffb640 1 -- 172.21.15.186:0/3757235604 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fbdc008ae80 con 0x7fbdb0004d40 2024-03-28T15:01:28.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.409+0000 7fbdc8b15640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbda806c530 0x7fbda806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.410+0000 7fbdc8b15640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbda806c530 0x7fbda806e9f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4153fb0 tx=0x7fbdb8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:28.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.414+0000 7fbdb5ffb640 1 -- 172.21.15.186:0/3757235604 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fbdc4150a10 con 0x7fbdb0004d40 2024-03-28T15:01:28.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.589+0000 7fbdcb57a640 1 -- 172.21.15.186:0/3757235604 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbdc40c8030 con 0x7fbdb0004d40 2024-03-28T15:01:28.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.590+0000 7fbdb5ffb640 1 -- 172.21.15.186:0/3757235604 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12920 (secure 0 0 0) 0x7fbdc40c8030 con 0x7fbdb0004d40 2024-03-28T15:01:28.590 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:28.591 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":16,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:21.561039+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:28.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.592+0000 7fbdcb57a640 1 -- 172.21.15.186:0/3757235604 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbda806c530 msgr2=0x7fbda806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.592+0000 7fbdcb57a640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbda806c530 0x7fbda806e9f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4153fb0 tx=0x7fbdb8008040 comp rx=0 tx=0).stop 2024-03-28T15:01:28.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.592+0000 7fbdcb57a640 1 -- 172.21.15.186:0/3757235604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 msgr2=0x7fbdc4153900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.592+0000 7fbdcb57a640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 0x7fbdc4153900 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0007340 tx=0x7fbdc0007800 comp rx=0 tx=0).stop 2024-03-28T15:01:28.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.592+0000 7fbdcb57a640 1 -- 172.21.15.186:0/3757235604 shutdown_connections 2024-03-28T15:01:28.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.592+0000 7fbdcb57a640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbda806c530 0x7fbda806e9f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.592+0000 7fbdcb57a640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdc40c59e0 0x7fbdc40c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.592+0000 7fbdcb57a640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdc414c630 0x7fbdc4159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.592+0000 7fbdcb57a640 1 --2- 172.21.15.186:0/3757235604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdb0004d40 0x7fbdc4153900 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.592+0000 7fbdcb57a640 1 -- 172.21.15.186:0/3757235604 >> 172.21.15.186:0/3757235604 conn(0x7fbdc40cc580 msgr2=0x7fbdc414f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:28.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.592+0000 7fbdcb57a640 1 -- 172.21.15.186:0/3757235604 shutdown_connections 2024-03-28T15:01:28.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.593+0000 7fbdcb57a640 1 -- 172.21.15.186:0/3757235604 wait complete. 2024-03-28T15:01:28.603 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:01:18.661274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:01:28.604 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:01:28.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.707+0000 7fa743fd7640 1 Processor -- start 2024-03-28T15:01:28.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.707+0000 7fa743fd7640 1 -- start start 2024-03-28T15:01:28.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa743fd7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa73c070210 0x7fa73c0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa743fd7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa73c0c7060 0x7fa73c0ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa743fd7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa73c15a8d0 con 0x7fa73c154bd0 2024-03-28T15:01:28.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa743fd7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa73c15b1e0 con 0x7fa73c0c7060 2024-03-28T15:01:28.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa743fd7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa73c062ad0 con 0x7fa73c070210 2024-03-28T15:01:28.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa742574640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa73c0c7060 0x7fa73c0ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa742574640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa73c0c7060 0x7fa73c0ccb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40986/0 (socket says 172.21.15.186:40986) 2024-03-28T15:01:28.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa742574640 1 -- 172.21.15.186:0/703582313 learned_addr learned my addr 172.21.15.186:0/703582313 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:28.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa741572640 1 --2- 172.21.15.186:0/703582313 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa73c070210 0x7fa73c0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa742574640 1 -- 172.21.15.186:0/703582313 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa73c070210 msgr2=0x7fa73c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:28.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa742574640 1 --2- 172.21.15.186:0/703582313 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa73c070210 0x7fa73c0cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa742574640 1 -- 172.21.15.186:0/703582313 >> v1:172.21.15.191:6789/0 conn(0x7fa73c154bd0 legacy=0x7fa73c0c8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.708+0000 7fa742574640 1 -- 172.21.15.186:0/703582313 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa73c0c7d30 con 0x7fa73c0c7060 2024-03-28T15:01:28.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.709+0000 7fa742574640 1 --2- 172.21.15.186:0/703582313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa73c0c7060 0x7fa73c0ccb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa738009870 tx=0x7fa73802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c899269f5b0be536 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:28.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.709+0000 7fa740d71640 1 -- 172.21.15.186:0/703582313 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738034070 con 0x7fa73c0c7060 2024-03-28T15:01:28.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.709+0000 7fa740d71640 1 -- 172.21.15.186:0/703582313 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa738030840 con 0x7fa73c0c7060 2024-03-28T15:01:28.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.709+0000 7fa740d71640 1 -- 172.21.15.186:0/703582313 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738030c10 con 0x7fa73c0c7060 2024-03-28T15:01:28.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.710+0000 7fa743fd7640 1 -- 172.21.15.186:0/703582313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa73c0c7060 msgr2=0x7fa73c0ccb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.710+0000 7fa743fd7640 1 --2- 172.21.15.186:0/703582313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa73c0c7060 0x7fa73c0ccb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa738009870 tx=0x7fa73802b980 comp rx=0 tx=0).stop 2024-03-28T15:01:28.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.710+0000 7fa743fd7640 1 -- 172.21.15.186:0/703582313 shutdown_connections 2024-03-28T15:01:28.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.710+0000 7fa743fd7640 1 --2- 172.21.15.186:0/703582313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa73c0c7060 0x7fa73c0ccb30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.710+0000 7fa743fd7640 1 --2- 172.21.15.186:0/703582313 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa73c070210 0x7fa73c0cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.710+0000 7fa743fd7640 1 -- 172.21.15.186:0/703582313 >> 172.21.15.186:0/703582313 conn(0x7fa73c06df10 msgr2=0x7fa73c06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:28.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.710+0000 7fa743fd7640 1 -- 172.21.15.186:0/703582313 shutdown_connections 2024-03-28T15:01:28.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.710+0000 7fa743fd7640 1 -- 172.21.15.186:0/703582313 wait complete. 2024-03-28T15:01:28.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.711+0000 7fa743fd7640 1 Processor -- start 2024-03-28T15:01:28.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.711+0000 7fa743fd7640 1 -- start start 2024-03-28T15:01:28.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa743fd7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa73c070210 0x7fa73c0c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa743fd7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa73c0c7060 0x7fa73c1f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa743fd7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa73c154bd0 0x7fa73c1f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa743fd7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa73c15ae70 con 0x7fa73c154bd0 2024-03-28T15:01:28.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa743fd7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa73c15ab90 con 0x7fa73c0c7060 2024-03-28T15:01:28.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa743fd7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa73c154fb0 con 0x7fa73c070210 2024-03-28T15:01:28.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa742574640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa73c154bd0 0x7fa73c1f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa742574640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa73c154bd0 0x7fa73c1f2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40996/0 (socket says 172.21.15.186:40996) 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa742574640 1 -- 172.21.15.186:0/2131314665 learned_addr learned my addr 172.21.15.186:0/2131314665 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa741572640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa73c0c7060 0x7fa73c1f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa741d73640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa73c070210 0x7fa73c0c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa742574640 1 -- 172.21.15.186:0/2131314665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa73c070210 msgr2=0x7fa73c0c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa742574640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa73c070210 0x7fa73c0c5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa742574640 1 -- 172.21.15.186:0/2131314665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa73c0c7060 msgr2=0x7fa73c1f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa742574640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa73c0c7060 0x7fa73c1f2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa742574640 1 -- 172.21.15.186:0/2131314665 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa73c0cb360 con 0x7fa73c154bd0 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa741d73640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa73c070210 0x7fa73c0c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa742574640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa73c154bd0 0x7fa73c1f2b60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa738009870 tx=0x7fa738004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.712+0000 7fa7327fc640 1 -- 172.21.15.186:0/2131314665 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738048070 con 0x7fa73c154bd0 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.713+0000 7fa7327fc640 1 -- 172.21.15.186:0/2131314665 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa738030640 con 0x7fa73c154bd0 2024-03-28T15:01:28.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.713+0000 7fa743fd7640 1 -- 172.21.15.186:0/2131314665 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa73c0cb640 con 0x7fa73c154bd0 2024-03-28T15:01:28.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.713+0000 7fa7327fc640 1 -- 172.21.15.186:0/2131314665 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738034040 con 0x7fa73c154bd0 2024-03-28T15:01:28.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.713+0000 7fa743fd7640 1 -- 172.21.15.186:0/2131314665 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa73c1f1b40 con 0x7fa73c154bd0 2024-03-28T15:01:28.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.714+0000 7fa743fd7640 1 -- 172.21.15.186:0/2131314665 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa73c069570 con 0x7fa73c154bd0 2024-03-28T15:01:28.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.714+0000 7fa7327fc640 1 -- 172.21.15.186:0/2131314665 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa73804b070 con 0x7fa73c154bd0 2024-03-28T15:01:28.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.714+0000 7fa7327fc640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa72006c560 0x7fa72006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:28.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.714+0000 7fa7327fc640 1 -- 172.21.15.186:0/2131314665 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fa7380ad1b0 con 0x7fa73c154bd0 2024-03-28T15:01:28.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.714+0000 7fa741d73640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa72006c560 0x7fa72006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:28.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.715+0000 7fa741d73640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa72006c560 0x7fa72006ea20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa734001d60 tx=0x7fa734009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:28.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.719+0000 7fa7327fc640 1 -- 172.21.15.186:0/2131314665 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa73807b4d0 con 0x7fa73c154bd0 2024-03-28T15:01:28.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.895+0000 7fa743fd7640 1 -- 172.21.15.186:0/2131314665 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa73c062c50 con 0x7fa73c154bd0 2024-03-28T15:01:28.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.895+0000 7fa7327fc640 1 -- 172.21.15.186:0/2131314665 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fa73807f7d0 con 0x7fa73c154bd0 2024-03-28T15:01:28.895 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 1 2024-03-28T15:01:28.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 -- 172.21.15.186:0/2131314665 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa72006c560 msgr2=0x7fa72006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa72006c560 0x7fa72006ea20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa734001d60 tx=0x7fa734009040 comp rx=0 tx=0).stop 2024-03-28T15:01:28.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 -- 172.21.15.186:0/2131314665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa73c154bd0 msgr2=0x7fa73c1f2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:28.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa73c154bd0 0x7fa73c1f2b60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa738009870 tx=0x7fa738004730 comp rx=0 tx=0).stop 2024-03-28T15:01:28.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 -- 172.21.15.186:0/2131314665 shutdown_connections 2024-03-28T15:01:28.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa72006c560 0x7fa72006ea20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa73c154bd0 0x7fa73c1f2b60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa73c0c7060 0x7fa73c1f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 --2- 172.21.15.186:0/2131314665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa73c070210 0x7fa73c0c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:28.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 -- 172.21.15.186:0/2131314665 >> 172.21.15.186:0/2131314665 conn(0x7fa73c06df10 msgr2=0x7fa73c152e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:28.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 -- 172.21.15.186:0/2131314665 shutdown_connections 2024-03-28T15:01:28.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:28.897+0000 7fa743fd7640 1 -- 172.21.15.186:0/2131314665 wait complete. 2024-03-28T15:01:28.908 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T15:01:29.009 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.008+0000 7f7d56f20640 1 Processor -- start 2024-03-28T15:01:29.009 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.008+0000 7f7d56f20640 1 -- start start 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.009+0000 7f7d56f20640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d500cb560 0x7f7d50166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.009+0000 7f7d56f20640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d500c6f90 con 0x7f7d500c5380 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.009+0000 7f7d56f20640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d500cc970 con 0x7f7d500caa70 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.009+0000 7f7d56f20640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7d50071a80 con 0x7f7d500cb560 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.009+0000 7f7d4ffff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f7d500c5380 0x7f7d500cae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:37362/0 (socket says 172.21.15.186:37362) 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.009+0000 7f7d4ffff640 1 -- 172.21.15.186:0/825879770 learned_addr learned my addr 172.21.15.186:0/825879770 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.009+0000 7f7d554bd640 1 --2- 172.21.15.186:0/825879770 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d500cb560 0x7f7d50166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.010+0000 7f7d4f7fe640 1 -- 172.21.15.186:0/825879770 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 944871851 0 0) 0x7f7d44001110 con 0x7f7d500c5380 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.010+0000 7f7d4f7fe640 1 -- 172.21.15.186:0/825879770 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d38003520 con 0x7f7d500c5380 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.010+0000 7f7d4f7fe640 1 -- 172.21.15.186:0/825879770 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 139327540 0 0) 0x7f7d500cc970 con 0x7f7d500caa70 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.010+0000 7f7d554bd640 1 -- 172.21.15.186:0/825879770 >> v1:172.21.15.191:6789/0 conn(0x7f7d500caa70 legacy=0x7f7d500c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.010+0000 7f7d554bd640 1 -- 172.21.15.186:0/825879770 >> v1:172.21.15.186:6789/0 conn(0x7f7d500c5380 legacy=0x7f7d500cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:29.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.010+0000 7f7d554bd640 1 -- 172.21.15.186:0/825879770 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d500cc690 con 0x7f7d500cb560 2024-03-28T15:01:29.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.010+0000 7f7d554bd640 1 --2- 172.21.15.186:0/825879770 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d500cb560 0x7f7d50166230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7d500ccb40 tx=0x7f7d4802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3283314b3c269970 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:29.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.011+0000 7f7d4f7fe640 1 -- 172.21.15.186:0/825879770 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d48039070 con 0x7f7d500cb560 2024-03-28T15:01:29.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.011+0000 7f7d4f7fe640 1 -- 172.21.15.186:0/825879770 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d4802ccd0 con 0x7f7d500cb560 2024-03-28T15:01:29.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.011+0000 7f7d4f7fe640 1 -- 172.21.15.186:0/825879770 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d48034660 con 0x7f7d500cb560 2024-03-28T15:01:29.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.011+0000 7f7d56f20640 1 -- 172.21.15.186:0/825879770 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d500cb560 msgr2=0x7f7d50166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:29.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.011+0000 7f7d56f20640 1 --2- 172.21.15.186:0/825879770 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d500cb560 0x7f7d50166230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7d500ccb40 tx=0x7f7d4802bf20 comp rx=0 tx=0).stop 2024-03-28T15:01:29.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.011+0000 7f7d56f20640 1 -- 172.21.15.186:0/825879770 shutdown_connections 2024-03-28T15:01:29.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.011+0000 7f7d56f20640 1 --2- 172.21.15.186:0/825879770 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d500cb560 0x7f7d50166230 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.011+0000 7f7d56f20640 1 -- 172.21.15.186:0/825879770 >> 172.21.15.186:0/825879770 conn(0x7f7d500d00f0 msgr2=0x7f7d500d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:29.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.011+0000 7f7d56f20640 1 -- 172.21.15.186:0/825879770 shutdown_connections 2024-03-28T15:01:29.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.011+0000 7f7d56f20640 1 -- 172.21.15.186:0/825879770 wait complete. 2024-03-28T15:01:29.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.012+0000 7f7d56f20640 1 Processor -- start 2024-03-28T15:01:29.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.012+0000 7f7d56f20640 1 -- start start 2024-03-28T15:01:29.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.013+0000 7f7d56f20640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d500c5380 0x7f7d5014cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.013+0000 7f7d56f20640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d500caa70 0x7f7d5014eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.013+0000 7f7d56f20640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d500cb560 0x7f7d500c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.013+0000 7f7d56f20640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d500c5cc0 con 0x7f7d500c5380 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.013+0000 7f7d56f20640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7d500c6030 con 0x7f7d500caa70 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.013+0000 7f7d56f20640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7d5015b810 con 0x7f7d500cb560 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.013+0000 7f7d54cbc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d500c5380 0x7f7d5014cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.013+0000 7f7d54cbc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d500c5380 0x7f7d5014cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41012/0 (socket says 172.21.15.186:41012) 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.013+0000 7f7d54cbc640 1 -- 172.21.15.186:0/4090445338 learned_addr learned my addr 172.21.15.186:0/4090445338 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.013+0000 7f7d554bd640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d500cb560 0x7f7d500c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.013+0000 7f7d4ffff640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d500caa70 0x7f7d5014eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d54cbc640 1 -- 172.21.15.186:0/4090445338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d500cb560 msgr2=0x7f7d500c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d54cbc640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d500cb560 0x7f7d500c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d54cbc640 1 -- 172.21.15.186:0/4090445338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d500caa70 msgr2=0x7f7d5014eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d54cbc640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d500caa70 0x7f7d5014eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d54cbc640 1 -- 172.21.15.186:0/4090445338 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d500c8d10 con 0x7f7d500c5380 2024-03-28T15:01:29.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d4ffff640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d500caa70 0x7f7d5014eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:29.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d54cbc640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d500c5380 0x7f7d5014cc10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000c4e0 tx=0x7f7d4000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:29.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d4d7fa640 1 -- 172.21.15.186:0/4090445338 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40017070 con 0x7f7d500c5380 2024-03-28T15:01:29.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d4d7fa640 1 -- 172.21.15.186:0/4090445338 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d400035f0 con 0x7f7d500c5380 2024-03-28T15:01:29.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d56f20640 1 -- 172.21.15.186:0/4090445338 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d5014f0d0 con 0x7f7d500c5380 2024-03-28T15:01:29.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d4d7fa640 1 -- 172.21.15.186:0/4090445338 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d400039e0 con 0x7f7d500c5380 2024-03-28T15:01:29.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.014+0000 7f7d56f20640 1 -- 172.21.15.186:0/4090445338 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d501f1390 con 0x7f7d500c5380 2024-03-28T15:01:29.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.015+0000 7f7d56f20640 1 -- 172.21.15.186:0/4090445338 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d500686d0 con 0x7f7d500c5380 2024-03-28T15:01:29.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.015+0000 7f7d4d7fa640 1 -- 172.21.15.186:0/4090445338 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7d4001a070 con 0x7f7d500c5380 2024-03-28T15:01:29.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.016+0000 7f7d4d7fa640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7d3c06c560 0x7f7d3c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:29.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.016+0000 7f7d4ffff640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7d3c06c560 0x7f7d3c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:29.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.016+0000 7f7d4d7fa640 1 -- 172.21.15.186:0/4090445338 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f7d4008a5b0 con 0x7f7d500c5380 2024-03-28T15:01:29.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.016+0000 7f7d4ffff640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7d3c06c560 0x7f7d3c06ea20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d44007fd0 tx=0x7f7d44009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:29.020 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.020+0000 7f7d4d7fa640 1 -- 172.21.15.186:0/4090445338 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7d40058960 con 0x7f7d500c5380 2024-03-28T15:01:29.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.198+0000 7f7d56f20640 1 -- 172.21.15.186:0/4090445338 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7d500b3b10 con 0x7f7d500c5380 2024-03-28T15:01:29.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.198+0000 7f7d4d7fa640 1 -- 172.21.15.186:0/4090445338 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f7d4005cc60 con 0x7f7d500c5380 2024-03-28T15:01:29.199 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:29.199 INFO:teuthology.orchestra.run.smithi186.stdout:[ 2024-03-28T15:01:29.199 INFO:teuthology.orchestra.run.smithi186.stdout: { 2024-03-28T15:01:29.199 INFO:teuthology.orchestra.run.smithi186.stdout: "poolnum": 1, 2024-03-28T15:01:29.199 INFO:teuthology.orchestra.run.smithi186.stdout: "poolname": ".mgr" 2024-03-28T15:01:29.199 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-03-28T15:01:29.199 INFO:teuthology.orchestra.run.smithi186.stdout:] 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 -- 172.21.15.186:0/4090445338 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7d3c06c560 msgr2=0x7f7d3c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7d3c06c560 0x7f7d3c06ea20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d44007fd0 tx=0x7f7d44009040 comp rx=0 tx=0).stop 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 -- 172.21.15.186:0/4090445338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d500c5380 msgr2=0x7f7d5014cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d500c5380 0x7f7d5014cc10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000c4e0 tx=0x7f7d4000c9a0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 -- 172.21.15.186:0/4090445338 shutdown_connections 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7d3c06c560 0x7f7d3c06ea20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d500cb560 0x7f7d500c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d500caa70 0x7f7d5014eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 --2- 172.21.15.186:0/4090445338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d500c5380 0x7f7d5014cc10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 -- 172.21.15.186:0/4090445338 >> 172.21.15.186:0/4090445338 conn(0x7f7d500d00f0 msgr2=0x7f7d500cae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 -- 172.21.15.186:0/4090445338 shutdown_connections 2024-03-28T15:01:29.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.200+0000 7f7d56f20640 1 -- 172.21.15.186:0/4090445338 wait complete. 2024-03-28T15:01:29.211 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T15:01:29.211 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T15:01:29.317 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.317+0000 7f9340c76640 1 Processor -- start 2024-03-28T15:01:29.317 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.317+0000 7f9340c76640 1 -- start start 2024-03-28T15:01:29.319 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.318+0000 7f9340c76640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f933c154ae0 0x7f933c0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:29.319 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.318+0000 7f9340c76640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f933c0c52f0 0x7f933c15b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:29.319 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.318+0000 7f9340c76640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933c0cb120 con 0x7f933c15e820 2024-03-28T15:01:29.319 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.318+0000 7f9340c76640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f933c0cb400 con 0x7f933c154ae0 2024-03-28T15:01:29.319 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.318+0000 7f9340c76640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f933c15a930 con 0x7f933c0c52f0 2024-03-28T15:01:29.319 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.318+0000 7f933a59c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f933c154ae0 0x7f933c0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:29.319 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.318+0000 7f9339d9b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f933c0c52f0 0x7f933c15b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:29.319 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.318+0000 7f9339d9b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f933c0c52f0 0x7f933c15b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:56652/0 (socket says 172.21.15.186:56652) 2024-03-28T15:01:29.319 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.318+0000 7f933a59c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f933c154ae0 0x7f933c0c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:57946/0 (socket says 172.21.15.186:57946) 2024-03-28T15:01:29.319 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.318+0000 7f9339d9b640 1 -- 172.21.15.186:0/2948687725 learned_addr learned my addr 172.21.15.186:0/2948687725 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:29.319 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.319+0000 7f933a59c640 1 -- 172.21.15.186:0/2948687725 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f933c0c52f0 msgr2=0x7f933c15b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:29.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.319+0000 7f933a59c640 1 --2- 172.21.15.186:0/2948687725 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f933c0c52f0 0x7f933c15b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.319+0000 7f933a59c640 1 -- 172.21.15.186:0/2948687725 >> v1:172.21.15.191:6789/0 conn(0x7f933c15e820 legacy=0x7f933c0ccaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:29.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.319+0000 7f933959a640 1 -- 172.21.15.186:0/2948687725 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872553568 0 0) 0x7f933c0cb120 con 0x7f933c15e820 2024-03-28T15:01:29.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.319+0000 7f933a59c640 1 -- 172.21.15.186:0/2948687725 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933c15ac10 con 0x7f933c154ae0 2024-03-28T15:01:29.321 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f933a59c640 1 --2- 172.21.15.186:0/2948687725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f933c154ae0 0x7f933c0c5840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f932c009dc0 tx=0x7f932c02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e24322abc22e200 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:29.321 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f933959a640 1 -- 172.21.15.186:0/2948687725 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c039070 con 0x7f933c154ae0 2024-03-28T15:01:29.321 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f933959a640 1 -- 172.21.15.186:0/2948687725 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f932c02cb20 con 0x7f933c154ae0 2024-03-28T15:01:29.321 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f933959a640 1 -- 172.21.15.186:0/2948687725 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c034440 con 0x7f933c154ae0 2024-03-28T15:01:29.321 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f9340c76640 1 -- 172.21.15.186:0/2948687725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f933c154ae0 msgr2=0x7f933c0c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:29.321 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f9340c76640 1 --2- 172.21.15.186:0/2948687725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f933c154ae0 0x7f933c0c5840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f932c009dc0 tx=0x7f932c02bd70 comp rx=0 tx=0).stop 2024-03-28T15:01:29.321 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f9340c76640 1 -- 172.21.15.186:0/2948687725 shutdown_connections 2024-03-28T15:01:29.322 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f9340c76640 1 --2- 172.21.15.186:0/2948687725 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f933c0c52f0 0x7f933c15b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.322 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f9340c76640 1 --2- 172.21.15.186:0/2948687725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f933c154ae0 0x7f933c0c5840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.322 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f9340c76640 1 -- 172.21.15.186:0/2948687725 >> 172.21.15.186:0/2948687725 conn(0x7f933c073d40 msgr2=0x7f933c06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:29.322 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f9340c76640 1 -- 172.21.15.186:0/2948687725 shutdown_connections 2024-03-28T15:01:29.322 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.320+0000 7f9340c76640 1 -- 172.21.15.186:0/2948687725 wait complete. 2024-03-28T15:01:29.324 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f9340c76640 1 Processor -- start 2024-03-28T15:01:29.324 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f9340c76640 1 -- start start 2024-03-28T15:01:29.324 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f9340c76640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f933c0c52f0 0x7f933c1f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:29.324 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f9340c76640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f933c154ae0 0x7f933c1f0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:29.324 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f9340c76640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f933c15e820 0x7f933c0c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:29.324 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f9340c76640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f933c0cb6e0 con 0x7f933c0c52f0 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f9340c76640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f933c15ecb0 con 0x7f933c15e820 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f9340c76640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f933c0cd1b0 con 0x7f933c154ae0 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f933a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f933c0c52f0 0x7f933c1f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f933a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f933c0c52f0 0x7f933c1f4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41016/0 (socket says 172.21.15.186:41016) 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f933a59c640 1 -- 172.21.15.186:0/455110368 learned_addr learned my addr 172.21.15.186:0/455110368 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f9339d9b640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f933c154ae0 0x7f933c1f0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.322+0000 7f933ad9d640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f933c15e820 0x7f933c0c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.323+0000 7f933a59c640 1 -- 172.21.15.186:0/455110368 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f933c154ae0 msgr2=0x7f933c1f0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.323+0000 7f933a59c640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f933c154ae0 0x7f933c1f0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.323+0000 7f933a59c640 1 -- 172.21.15.186:0/455110368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f933c15e820 msgr2=0x7f933c0c98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.323+0000 7f933a59c640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f933c15e820 0x7f933c0c98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.323+0000 7f933a59c640 1 -- 172.21.15.186:0/455110368 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933c1f03f0 con 0x7f933c0c52f0 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.323+0000 7f933a59c640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f933c0c52f0 0x7f933c1f4160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f932c037040 tx=0x7f932c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:29.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.323+0000 7f932affd640 1 -- 172.21.15.186:0/455110368 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c039050 con 0x7f933c0c52f0 2024-03-28T15:01:29.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.323+0000 7f932affd640 1 -- 172.21.15.186:0/455110368 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f932c03a040 con 0x7f933c0c52f0 2024-03-28T15:01:29.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.323+0000 7f932affd640 1 -- 172.21.15.186:0/455110368 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c009450 con 0x7f933c0c52f0 2024-03-28T15:01:29.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.323+0000 7f9340c76640 1 -- 172.21.15.186:0/455110368 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f933c153310 con 0x7f933c0c52f0 2024-03-28T15:01:29.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.323+0000 7f9340c76640 1 -- 172.21.15.186:0/455110368 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f933c1538d0 con 0x7f933c0c52f0 2024-03-28T15:01:29.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.324+0000 7f9340c76640 1 -- 172.21.15.186:0/455110368 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f933c0c6bc0 con 0x7f933c0c52f0 2024-03-28T15:01:29.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.324+0000 7f932affd640 1 -- 172.21.15.186:0/455110368 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f932c047020 con 0x7f933c0c52f0 2024-03-28T15:01:29.327 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.325+0000 7f932affd640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f931c06c530 0x7f931c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:29.327 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.325+0000 7f9339d9b640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f931c06c530 0x7f931c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:29.327 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.325+0000 7f932affd640 1 -- 172.21.15.186:0/455110368 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f932c0ad4d0 con 0x7f933c0c52f0 2024-03-28T15:01:29.327 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.325+0000 7f9339d9b640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f931c06c530 0x7f931c06e9f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f933c1e6b90 tx=0x7f9330009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:29.329 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.329+0000 7f932affd640 1 -- 172.21.15.186:0/455110368 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f933c0c6bc0 con 0x7f933c0c52f0 2024-03-28T15:01:29.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:29.508+0000 7f9340c76640 1 -- 172.21.15.186:0/455110368 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f933c069570 con 0x7f933c0c52f0 2024-03-28T15:01:30.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.070+0000 7f932affd640 1 -- 172.21.15.186:0/455110368 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7f932c04e020 con 0x7f933c0c52f0 2024-03-28T15:01:30.070 INFO:teuthology.orchestra.run.smithi186.stderr:pool 'cephfs_metadata' created 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 -- 172.21.15.186:0/455110368 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f931c06c530 msgr2=0x7f931c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f931c06c530 0x7f931c06e9f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f933c1e6b90 tx=0x7f9330009240 comp rx=0 tx=0).stop 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 -- 172.21.15.186:0/455110368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f933c0c52f0 msgr2=0x7f933c1f4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f933c0c52f0 0x7f933c1f4160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f932c037040 tx=0x7f932c004a40 comp rx=0 tx=0).stop 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 -- 172.21.15.186:0/455110368 shutdown_connections 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f931c06c530 0x7f931c06e9f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f933c15e820 0x7f933c0c98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f933c154ae0 0x7f933c1f0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 --2- 172.21.15.186:0/455110368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f933c0c52f0 0x7f933c1f4160 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 -- 172.21.15.186:0/455110368 >> 172.21.15.186:0/455110368 conn(0x7f933c073d40 msgr2=0x7f933c06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 -- 172.21.15.186:0/455110368 shutdown_connections 2024-03-28T15:01:30.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.072+0000 7f9340c76640 1 -- 172.21.15.186:0/455110368 wait complete. 2024-03-28T15:01:30.088 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T15:01:30.197 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.197+0000 7f5bf837a640 1 Processor -- start 2024-03-28T15:01:30.197 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.197+0000 7f5bf837a640 1 -- start start 2024-03-28T15:01:30.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf837a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5bf006ee50 0x7f5bf006df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:30.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf837a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bf0159180 0x7f5bf01525a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:30.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf837a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bf0072370 con 0x7f5bf006f230 2024-03-28T15:01:30.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf837a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f5bf00c87b0 con 0x7f5bf006ee50 2024-03-28T15:01:30.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf837a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5bf0071ee0 con 0x7f5bf0159180 2024-03-28T15:01:30.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf6116640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5bf006ee50 0x7f5bf006df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:30.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf6116640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5bf006ee50 0x7f5bf006df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41020/0 (socket says 172.21.15.186:41020) 2024-03-28T15:01:30.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf6116640 1 -- 172.21.15.186:0/4017760221 learned_addr learned my addr 172.21.15.186:0/4017760221 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:30.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf6917640 1 --2- 172.21.15.186:0/4017760221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bf0159180 0x7f5bf01525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:30.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf6116640 1 -- 172.21.15.186:0/4017760221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bf0159180 msgr2=0x7f5bf01525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:30.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf6116640 1 --2- 172.21.15.186:0/4017760221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bf0159180 0x7f5bf01525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:30.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf6116640 1 -- 172.21.15.186:0/4017760221 >> v1:172.21.15.191:6789/0 conn(0x7f5bf006f230 legacy=0x7f5bf006c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:30.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.198+0000 7f5bf6116640 1 -- 172.21.15.186:0/4017760221 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf006e550 con 0x7f5bf006ee50 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.199+0000 7f5bf6116640 1 --2- 172.21.15.186:0/4017760221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bf006ee50 0x7f5bf006df30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009e50 tx=0x7f5bec02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f23ca8c5137c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.199+0000 7f5bf5114640 1 -- 172.21.15.186:0/4017760221 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec02cb00 con 0x7f5bf006ee50 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.199+0000 7f5bf5114640 1 -- 172.21.15.186:0/4017760221 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bec02cca0 con 0x7f5bf006ee50 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.199+0000 7f5bf5114640 1 -- 172.21.15.186:0/4017760221 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec034610 con 0x7f5bf006ee50 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.200+0000 7f5bf837a640 1 -- 172.21.15.186:0/4017760221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bf006ee50 msgr2=0x7f5bf006df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.200+0000 7f5bf837a640 1 --2- 172.21.15.186:0/4017760221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bf006ee50 0x7f5bf006df30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009e50 tx=0x7f5bec02bf60 comp rx=0 tx=0).stop 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.200+0000 7f5bf837a640 1 -- 172.21.15.186:0/4017760221 shutdown_connections 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.200+0000 7f5bf837a640 1 --2- 172.21.15.186:0/4017760221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bf0159180 0x7f5bf01525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.200+0000 7f5bf837a640 1 --2- 172.21.15.186:0/4017760221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bf006ee50 0x7f5bf006df30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.200+0000 7f5bf837a640 1 -- 172.21.15.186:0/4017760221 >> 172.21.15.186:0/4017760221 conn(0x7f5bf006c010 msgr2=0x7f5bf00cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.200+0000 7f5bf837a640 1 -- 172.21.15.186:0/4017760221 shutdown_connections 2024-03-28T15:01:30.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.200+0000 7f5bf837a640 1 -- 172.21.15.186:0/4017760221 wait complete. 2024-03-28T15:01:30.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.201+0000 7f5bf837a640 1 Processor -- start 2024-03-28T15:01:30.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.201+0000 7f5bf837a640 1 -- start start 2024-03-28T15:01:30.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf837a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bf006ee50 0x7f5bf0154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:30.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf837a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bf006f230 0x7f5bf0155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:30.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf837a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bf0159180 0x7f5bf01e1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:30.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf837a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf0071890 con 0x7f5bf0159180 2024-03-28T15:01:30.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf837a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf00715b0 con 0x7f5bf006f230 2024-03-28T15:01:30.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf837a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5bf0071b70 con 0x7f5bf006ee50 2024-03-28T15:01:30.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf6917640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bf0159180 0x7f5bf01e1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:30.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf6917640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bf0159180 0x7f5bf01e1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41034/0 (socket says 172.21.15.186:41034) 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf6917640 1 -- 172.21.15.186:0/2580003510 learned_addr learned my addr 172.21.15.186:0/2580003510 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf5915640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bf006f230 0x7f5bf0155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf6116640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bf006ee50 0x7f5bf0154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf6917640 1 -- 172.21.15.186:0/2580003510 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bf006ee50 msgr2=0x7f5bf0154dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf6917640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bf006ee50 0x7f5bf0154dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf6917640 1 -- 172.21.15.186:0/2580003510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bf006f230 msgr2=0x7f5bf0155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf6917640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bf006f230 0x7f5bf0155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.202+0000 7f5bf6917640 1 -- 172.21.15.186:0/2580003510 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf01535b0 con 0x7f5bf0159180 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.203+0000 7f5bf6917640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bf0159180 0x7f5bf01e1ce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5be800d830 tx=0x7f5be800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.203+0000 7f5be6ffd640 1 -- 172.21.15.186:0/2580003510 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be800b6c0 con 0x7f5bf0159180 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.203+0000 7f5be6ffd640 1 -- 172.21.15.186:0/2580003510 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5be8019070 con 0x7f5bf0159180 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.203+0000 7f5bf837a640 1 -- 172.21.15.186:0/2580003510 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf0153850 con 0x7f5bf0159180 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.203+0000 7f5be6ffd640 1 -- 172.21.15.186:0/2580003510 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be800bac0 con 0x7f5bf0159180 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.203+0000 7f5bf837a640 1 -- 172.21.15.186:0/2580003510 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf0153e10 con 0x7f5bf0159180 2024-03-28T15:01:30.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.204+0000 7f5bf837a640 1 -- 172.21.15.186:0/2580003510 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bf0069570 con 0x7f5bf0159180 2024-03-28T15:01:30.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.204+0000 7f5be6ffd640 1 -- 172.21.15.186:0/2580003510 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5be801e460 con 0x7f5bf0159180 2024-03-28T15:01:30.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.204+0000 7f5be6ffd640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bd406c560 0x7f5bd406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:30.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.205+0000 7f5be6ffd640 1 -- 172.21.15.186:0/2580003510 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f5be808ba10 con 0x7f5bf0159180 2024-03-28T15:01:30.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.205+0000 7f5bf6116640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bd406c560 0x7f5bd406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:30.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.205+0000 7f5bf6116640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bd406c560 0x7f5bd406ea20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bec035000 tx=0x7f5bec030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:30.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.209+0000 7f5be6ffd640 1 -- 172.21.15.186:0/2580003510 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5be8059bd0 con 0x7f5bf0159180 2024-03-28T15:01:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:30.384+0000 7f5bf837a640 1 -- 172.21.15.186:0/2580003510 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f5bf00c9fd0 con 0x7f5bf0159180 2024-03-28T15:01:31.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.072+0000 7f5be6ffd640 1 -- 172.21.15.186:0/2580003510 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) ==== 170+0+0 (secure 0 0 0) 0x7f5be805ded0 con 0x7f5bf0159180 2024-03-28T15:01:31.072 INFO:teuthology.orchestra.run.smithi186.stderr:pool 'cephfs_data' created 2024-03-28T15:01:31.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.074+0000 7f5bf837a640 1 -- 172.21.15.186:0/2580003510 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bd406c560 msgr2=0x7f5bd406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:31.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.074+0000 7f5bf837a640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bd406c560 0x7f5bd406ea20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bec035000 tx=0x7f5bec030000 comp rx=0 tx=0).stop 2024-03-28T15:01:31.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.074+0000 7f5bf837a640 1 -- 172.21.15.186:0/2580003510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bf0159180 msgr2=0x7f5bf01e1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:31.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.074+0000 7f5bf837a640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bf0159180 0x7f5bf01e1ce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5be800d830 tx=0x7f5be800dcf0 comp rx=0 tx=0).stop 2024-03-28T15:01:31.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.075+0000 7f5bf837a640 1 -- 172.21.15.186:0/2580003510 shutdown_connections 2024-03-28T15:01:31.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.075+0000 7f5bf837a640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5bd406c560 0x7f5bd406ea20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:31.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.075+0000 7f5bf837a640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bf0159180 0x7f5bf01e1ce0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:31.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.075+0000 7f5bf837a640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bf006f230 0x7f5bf0155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:31.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.075+0000 7f5bf837a640 1 --2- 172.21.15.186:0/2580003510 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5bf006ee50 0x7f5bf0154dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:31.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.075+0000 7f5bf837a640 1 -- 172.21.15.186:0/2580003510 >> 172.21.15.186:0/2580003510 conn(0x7f5bf006c010 msgr2=0x7f5bf00c6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:31.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.075+0000 7f5bf837a640 1 -- 172.21.15.186:0/2580003510 shutdown_connections 2024-03-28T15:01:31.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.075+0000 7f5bf837a640 1 -- 172.21.15.186:0/2580003510 wait complete. 2024-03-28T15:01:31.096 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T15:01:31.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.200+0000 7fef3cbf7640 1 Processor -- start 2024-03-28T15:01:31.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.200+0000 7fef3cbf7640 1 -- start start 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.200+0000 7fef3cbf7640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fef3815c220 0x7fef3815c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.200+0000 7fef3cbf7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fef38158620 0x7fef38158a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.200+0000 7fef3cbf7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef38152d80 con 0x7fef381529a0 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.200+0000 7fef3cbf7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fef380c5190 con 0x7fef38158620 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.200+0000 7fef3cbf7640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fef380c3170 con 0x7fef3815c220 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.201+0000 7fef36d9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fef38158620 0x7fef38158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.201+0000 7fef36d9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fef38158620 0x7fef38158a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41050/0 (socket says 172.21.15.186:41050) 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.201+0000 7fef36d9d640 1 -- 172.21.15.186:0/1905880678 learned_addr learned my addr 172.21.15.186:0/1905880678 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.201+0000 7fef36d9d640 1 -- 172.21.15.186:0/1905880678 >> v2:172.21.15.191:3300/0 conn(0x7fef3815c220 msgr2=0x7fef3815c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.201+0000 7fef2ffff640 1 --2- 172.21.15.186:0/1905880678 >> v2:172.21.15.191:3300/0 conn(0x7fef3815c220 0x7fef3815c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.201+0000 7fef36d9d640 1 --2- 172.21.15.186:0/1905880678 >> v2:172.21.15.191:3300/0 conn(0x7fef3815c220 0x7fef3815c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:31.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.201+0000 7fef36d9d640 1 -- 172.21.15.186:0/1905880678 >> v1:172.21.15.186:6789/0 conn(0x7fef381529a0 legacy=0x7fef38070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:31.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.201+0000 7fef36d9d640 1 -- 172.21.15.186:0/1905880678 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef380c4bd0 con 0x7fef38158620 2024-03-28T15:01:31.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.201+0000 7fef35d9b640 1 -- 172.21.15.186:0/1905880678 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1108574340 0 0) 0x7fef38152d80 con 0x7fef381529a0 2024-03-28T15:01:31.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.201+0000 7fef2ffff640 1 --2- 172.21.15.186:0/1905880678 >> v2:172.21.15.191:3300/0 conn(0x7fef3815c220 0x7fef3815c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:31.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef36d9d640 1 --2- 172.21.15.186:0/1905880678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef38158620 0x7fef38158a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fef30009e50 tx=0x7fef3002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98dada65ad07bbd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:31.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef35d9b640 1 -- 172.21.15.186:0/1905880678 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3002ca80 con 0x7fef38158620 2024-03-28T15:01:31.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef35d9b640 1 -- 172.21.15.186:0/1905880678 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef3002cc20 con 0x7fef38158620 2024-03-28T15:01:31.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef35d9b640 1 -- 172.21.15.186:0/1905880678 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef300343f0 con 0x7fef38158620 2024-03-28T15:01:31.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef3cbf7640 1 -- 172.21.15.186:0/1905880678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef38158620 msgr2=0x7fef38158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:31.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef3cbf7640 1 --2- 172.21.15.186:0/1905880678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef38158620 0x7fef38158a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fef30009e50 tx=0x7fef3002bee0 comp rx=0 tx=0).stop 2024-03-28T15:01:31.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef3cbf7640 1 -- 172.21.15.186:0/1905880678 shutdown_connections 2024-03-28T15:01:31.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef3cbf7640 1 --2- 172.21.15.186:0/1905880678 >> v2:172.21.15.191:3300/0 conn(0x7fef3815c220 0x7fef3815c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:31.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef3cbf7640 1 --2- 172.21.15.186:0/1905880678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef38158620 0x7fef38158a70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:31.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef3cbf7640 1 -- 172.21.15.186:0/1905880678 >> 172.21.15.186:0/1905880678 conn(0x7fef38073610 msgr2=0x7fef3806d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:31.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef3cbf7640 1 -- 172.21.15.186:0/1905880678 shutdown_connections 2024-03-28T15:01:31.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.202+0000 7fef3cbf7640 1 -- 172.21.15.186:0/1905880678 wait complete. 2024-03-28T15:01:31.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.204+0000 7fef3cbf7640 1 Processor -- start 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.204+0000 7fef3cbf7640 1 -- start start 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.204+0000 7fef3cbf7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef381529a0 0x7fef381e2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.204+0000 7fef3cbf7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef38158620 0x7fef3813dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.204+0000 7fef3cbf7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef3815c220 0x7fef381efbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.204+0000 7fef3cbf7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef380c8ad0 con 0x7fef38158620 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.204+0000 7fef3cbf7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fef3815bd00 con 0x7fef381529a0 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.204+0000 7fef3cbf7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fef380c8c50 con 0x7fef3815c220 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.204+0000 7fef2ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef38158620 0x7fef3813dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.204+0000 7fef2ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef38158620 0x7fef3813dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41056/0 (socket says 172.21.15.186:41056) 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.204+0000 7fef2ffff640 1 -- 172.21.15.186:0/3073904977 learned_addr learned my addr 172.21.15.186:0/3073904977 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef36d9d640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef3815c220 0x7fef381efbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef3659c640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef381529a0 0x7fef381e2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef2ffff640 1 -- 172.21.15.186:0/3073904977 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef3815c220 msgr2=0x7fef381efbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:31.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef2ffff640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef3815c220 0x7fef381efbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef2ffff640 1 -- 172.21.15.186:0/3073904977 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef381529a0 msgr2=0x7fef381e2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef2ffff640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef381529a0 0x7fef381e2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef2ffff640 1 -- 172.21.15.186:0/3073904977 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef38071150 con 0x7fef38158620 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef36d9d640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef3815c220 0x7fef381efbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef3659c640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef381529a0 0x7fef381e2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef2ffff640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef38158620 0x7fef3813dc80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c7f0 tx=0x7fef2000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef2effd640 1 -- 172.21.15.186:0/3073904977 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef20015070 con 0x7fef38158620 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef2effd640 1 -- 172.21.15.186:0/3073904977 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef200077c0 con 0x7fef38158620 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef3cbf7640 1 -- 172.21.15.186:0/3073904977 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef381e2070 con 0x7fef38158620 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef2effd640 1 -- 172.21.15.186:0/3073904977 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef20007b70 con 0x7fef38158620 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.205+0000 7fef3cbf7640 1 -- 172.21.15.186:0/3073904977 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef381e1590 con 0x7fef38158620 2024-03-28T15:01:31.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.206+0000 7fef3cbf7640 1 -- 172.21.15.186:0/3073904977 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef380681e0 con 0x7fef38158620 2024-03-28T15:01:31.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.206+0000 7fef2effd640 1 -- 172.21.15.186:0/3073904977 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fef20018070 con 0x7fef38158620 2024-03-28T15:01:31.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.207+0000 7fef2effd640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef1806c530 0x7fef1806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:31.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.207+0000 7fef2effd640 1 -- 172.21.15.186:0/3073904977 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7fef2008b710 con 0x7fef38158620 2024-03-28T15:01:31.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.207+0000 7fef3659c640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef1806c530 0x7fef1806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:31.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.207+0000 7fef3659c640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef1806c530 0x7fef1806e9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef28000f60 tx=0x7fef28009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:31.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.211+0000 7fef2effd640 1 -- 172.21.15.186:0/3073904977 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fef20059e20 con 0x7fef38158620 2024-03-28T15:01:31.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:31.423+0000 7fef3cbf7640 1 -- 172.21.15.186:0/3073904977 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fef38069570 con 0x7fef38158620 2024-03-28T15:01:32.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.110+0000 7fef2effd640 1 -- 172.21.15.186:0/3073904977 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fef20059720 con 0x7fef38158620 2024-03-28T15:01:32.111 INFO:teuthology.orchestra.run.smithi186.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T15:01:32.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 -- 172.21.15.186:0/3073904977 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef1806c530 msgr2=0x7fef1806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef1806c530 0x7fef1806e9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef28000f60 tx=0x7fef28009040 comp rx=0 tx=0).stop 2024-03-28T15:01:32.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 -- 172.21.15.186:0/3073904977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef38158620 msgr2=0x7fef3813dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef38158620 0x7fef3813dc80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c7f0 tx=0x7fef2000ccb0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 -- 172.21.15.186:0/3073904977 shutdown_connections 2024-03-28T15:01:32.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef1806c530 0x7fef1806e9f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef3815c220 0x7fef381efbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef38158620 0x7fef3813dc80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 --2- 172.21.15.186:0/3073904977 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef381529a0 0x7fef381e2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 -- 172.21.15.186:0/3073904977 >> 172.21.15.186:0/3073904977 conn(0x7fef38073610 msgr2=0x7fef38070370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:32.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 -- 172.21.15.186:0/3073904977 shutdown_connections 2024-03-28T15:01:32.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.117+0000 7fef3cbf7640 1 -- 172.21.15.186:0/3073904977 wait complete. 2024-03-28T15:01:32.132 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:32.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.233+0000 7f65a6439640 1 Processor -- start 2024-03-28T15:01:32.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.233+0000 7f65a6439640 1 -- start start 2024-03-28T15:01:32.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f65a6439640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f65a00c53c0 0x7f65a00c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f65a6439640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a00c57e0 0x7f65a00cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f65a6439640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a015b080 con 0x7f65a0154bd0 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f65a6439640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f65a00c5ca0 con 0x7f65a00c53c0 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f65a6439640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f65a015b360 con 0x7f65a00c57e0 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f65a49d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a00c57e0 0x7f65a00cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f659ffff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f65a0154bd0 0x7f65a015a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:36044/0 (socket says 172.21.15.186:36044) 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f659ffff640 1 -- 172.21.15.186:0/841909696 learned_addr learned my addr 172.21.15.186:0/841909696 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f659f7fe640 1 --2- 172.21.15.186:0/841909696 >> v2:172.21.15.191:3300/0 conn(0x7f65a00c53c0 0x7f65a00c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f659f7fe640 1 -- 172.21.15.186:0/841909696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a00c57e0 msgr2=0x7f65a00cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f659f7fe640 1 --2- 172.21.15.186:0/841909696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a00c57e0 0x7f65a00cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f659f7fe640 1 -- 172.21.15.186:0/841909696 >> v1:172.21.15.191:6789/0 conn(0x7f65a0154bd0 legacy=0x7f65a015a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f659f7fe640 1 -- 172.21.15.186:0/841909696 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a015b640 con 0x7f65a00c53c0 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f659effd640 1 -- 172.21.15.186:0/841909696 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3936779870 0 0) 0x7f65a015b080 con 0x7f65a0154bd0 2024-03-28T15:01:32.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.234+0000 7f65a49d6640 1 --2- 172.21.15.186:0/841909696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a00c57e0 0x7f65a00cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.235+0000 7f659f7fe640 1 --2- 172.21.15.186:0/841909696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a00c53c0 0x7f65a00c66f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f65940097a0 tx=0x7f659402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ad45c696cae16c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.235+0000 7f659effd640 1 -- 172.21.15.186:0/841909696 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6594034070 con 0x7f65a00c53c0 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.235+0000 7f659effd640 1 -- 172.21.15.186:0/841909696 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6594030610 con 0x7f65a00c53c0 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.235+0000 7f659effd640 1 -- 172.21.15.186:0/841909696 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65940309e0 con 0x7f65a00c53c0 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.236+0000 7f65a6439640 1 -- 172.21.15.186:0/841909696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a00c53c0 msgr2=0x7f65a00c66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.236+0000 7f65a6439640 1 --2- 172.21.15.186:0/841909696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a00c53c0 0x7f65a00c66f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f65940097a0 tx=0x7f659402b960 comp rx=0 tx=0).stop 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.236+0000 7f65a6439640 1 -- 172.21.15.186:0/841909696 shutdown_connections 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.236+0000 7f65a6439640 1 --2- 172.21.15.186:0/841909696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a00c57e0 0x7f65a00cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.236+0000 7f65a6439640 1 --2- 172.21.15.186:0/841909696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a00c53c0 0x7f65a00c66f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.236+0000 7f65a6439640 1 -- 172.21.15.186:0/841909696 >> 172.21.15.186:0/841909696 conn(0x7f65a00731f0 msgr2=0x7f65a0073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.236+0000 7f65a6439640 1 -- 172.21.15.186:0/841909696 shutdown_connections 2024-03-28T15:01:32.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.236+0000 7f65a6439640 1 -- 172.21.15.186:0/841909696 wait complete. 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.237+0000 7f65a6439640 1 Processor -- start 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.237+0000 7f65a6439640 1 -- start start 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a6439640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a00c53c0 0x7f65a0153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a6439640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a00c57e0 0x7f65a01f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a6439640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65a0154bd0 0x7f65a014a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a6439640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65a00cc390 con 0x7f65a0154bd0 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a6439640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f65a00c61d0 con 0x7f65a00c57e0 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a6439640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f65a00c6c30 con 0x7f65a00c53c0 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a49d6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65a0154bd0 0x7f65a014a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a49d6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65a0154bd0 0x7f65a014a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41062/0 (socket says 172.21.15.186:41062) 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a49d6640 1 -- 172.21.15.186:0/3374922051 learned_addr learned my addr 172.21.15.186:0/3374922051 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f659f7fe640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a00c57e0 0x7f65a01f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f659ffff640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a00c53c0 0x7f65a0153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a49d6640 1 -- 172.21.15.186:0/3374922051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a00c53c0 msgr2=0x7f65a0153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a49d6640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a00c53c0 0x7f65a0153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a49d6640 1 -- 172.21.15.186:0/3374922051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a00c57e0 msgr2=0x7f65a01f0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a49d6640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a00c57e0 0x7f65a01f0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a49d6640 1 -- 172.21.15.186:0/3374922051 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a0158d40 con 0x7f65a0154bd0 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.238+0000 7f65a49d6640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65a0154bd0 0x7f65a014a870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6598009f00 tx=0x7f659800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.239+0000 7f659cff9640 1 -- 172.21.15.186:0/3374922051 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598018030 con 0x7f65a0154bd0 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.239+0000 7f659cff9640 1 -- 172.21.15.186:0/3374922051 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f659800ce50 con 0x7f65a0154bd0 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.239+0000 7f65a6439640 1 -- 172.21.15.186:0/3374922051 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a01f0800 con 0x7f65a0154bd0 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.239+0000 7f659cff9640 1 -- 172.21.15.186:0/3374922051 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598003bb0 con 0x7f65a0154bd0 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.239+0000 7f65a6439640 1 -- 172.21.15.186:0/3374922051 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a0158410 con 0x7f65a0154bd0 2024-03-28T15:01:32.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.240+0000 7f65a6439640 1 -- 172.21.15.186:0/3374922051 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a014b6c0 con 0x7f65a0154bd0 2024-03-28T15:01:32.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.240+0000 7f659cff9640 1 -- 172.21.15.186:0/3374922051 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6598005d10 con 0x7f65a0154bd0 2024-03-28T15:01:32.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.241+0000 7f659cff9640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f658c06c530 0x7f658c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.241+0000 7f659cff9640 1 -- 172.21.15.186:0/3374922051 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f659808c090 con 0x7f65a0154bd0 2024-03-28T15:01:32.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.241+0000 7f659ffff640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f658c06c530 0x7f658c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.241+0000 7f659ffff640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f658c06c530 0x7f658c06e9f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6590000f60 tx=0x7f6590007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:32.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.245+0000 7f659cff9640 1 -- 172.21.15.186:0/3374922051 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f65a014b6c0 con 0x7f65a0154bd0 2024-03-28T15:01:32.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.421+0000 7f65a6439640 1 -- 172.21.15.186:0/3374922051 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65a0069500 con 0x7f65a0154bd0 2024-03-28T15:01:32.422 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.422+0000 7f659cff9640 1 -- 172.21.15.186:0/3374922051 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16654 (secure 0 0 0) 0x7f6598022070 con 0x7f65a0154bd0 2024-03-28T15:01:32.422 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:32.423 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:31.424983+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 -- 172.21.15.186:0/3374922051 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f658c06c530 msgr2=0x7f658c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f658c06c530 0x7f658c06e9f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6590000f60 tx=0x7f6590007d50 comp rx=0 tx=0).stop 2024-03-28T15:01:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 -- 172.21.15.186:0/3374922051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65a0154bd0 msgr2=0x7f65a014a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65a0154bd0 0x7f65a014a870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6598009f00 tx=0x7f659800c490 comp rx=0 tx=0).stop 2024-03-28T15:01:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 -- 172.21.15.186:0/3374922051 shutdown_connections 2024-03-28T15:01:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f658c06c530 0x7f658c06e9f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65a0154bd0 0x7f65a014a870 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a00c57e0 0x7f65a01f0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 --2- 172.21.15.186:0/3374922051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a00c53c0 0x7f65a0153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 -- 172.21.15.186:0/3374922051 >> 172.21.15.186:0/3374922051 conn(0x7f65a00731f0 msgr2=0x7f65a00cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 -- 172.21.15.186:0/3374922051 shutdown_connections 2024-03-28T15:01:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.424+0000 7f65a6439640 1 -- 172.21.15.186:0/3374922051 wait complete. 2024-03-28T15:01:32.435 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:32.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.535+0000 7f550b257640 1 Processor -- start 2024-03-28T15:01:32.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.536+0000 7f550b257640 1 -- start start 2024-03-28T15:01:32.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.536+0000 7f550b257640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f55040c5380 0x7f55040cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.536+0000 7f550b257640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f55040cb390 0x7f5504165220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.536+0000 7f550b257640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55040c6f90 con 0x7f55040caa70 2024-03-28T15:01:32.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.536+0000 7f550b257640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f5504071a80 con 0x7f55040c5380 2024-03-28T15:01:32.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.536+0000 7f550b257640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f55040c6030 con 0x7f55040cb390 2024-03-28T15:01:32.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.536+0000 7f5503fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f55040c5380 0x7f55040cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.536+0000 7f5503fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f55040c5380 0x7f55040cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41074/0 (socket says 172.21.15.186:41074) 2024-03-28T15:01:32.537 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.536+0000 7f5503fff640 1 -- 172.21.15.186:0/87916622 learned_addr learned my addr 172.21.15.186:0/87916622 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:32.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.537+0000 7f55097f4640 1 --2- 172.21.15.186:0/87916622 >> v2:172.21.15.191:3300/0 conn(0x7f55040cb390 0x7f5504165220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.537+0000 7f5503fff640 1 -- 172.21.15.186:0/87916622 >> v2:172.21.15.191:3300/0 conn(0x7f55040cb390 msgr2=0x7f5504165220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:32.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.537+0000 7f5503fff640 1 --2- 172.21.15.186:0/87916622 >> v2:172.21.15.191:3300/0 conn(0x7f55040cb390 0x7f5504165220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.537+0000 7f5503fff640 1 -- 172.21.15.186:0/87916622 >> v1:172.21.15.186:6789/0 conn(0x7f55040caa70 legacy=0x7f55040c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.537+0000 7f5503fff640 1 -- 172.21.15.186:0/87916622 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55040cc970 con 0x7f55040c5380 2024-03-28T15:01:32.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.537+0000 7f55097f4640 1 --2- 172.21.15.186:0/87916622 >> v2:172.21.15.191:3300/0 conn(0x7f55040cb390 0x7f5504165220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:32.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.537+0000 7f5503fff640 1 --2- 172.21.15.186:0/87916622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55040c5380 0x7f55040cae50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f550408d380 tx=0x7f54f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1023e27207401d29 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:32.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.538+0000 7f55037fe640 1 -- 172.21.15.186:0/87916622 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f0034070 con 0x7f55040c5380 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.538+0000 7f55037fe640 1 -- 172.21.15.186:0/87916622 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f0030860 con 0x7f55040c5380 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.538+0000 7f55037fe640 1 -- 172.21.15.186:0/87916622 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f0030c30 con 0x7f55040c5380 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.538+0000 7f550b257640 1 -- 172.21.15.186:0/87916622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55040c5380 msgr2=0x7f55040cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.538+0000 7f550b257640 1 --2- 172.21.15.186:0/87916622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55040c5380 0x7f55040cae50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f550408d380 tx=0x7f54f002b9a0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.538+0000 7f550b257640 1 -- 172.21.15.186:0/87916622 shutdown_connections 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.538+0000 7f550b257640 1 --2- 172.21.15.186:0/87916622 >> v2:172.21.15.191:3300/0 conn(0x7f55040cb390 0x7f5504165220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.538+0000 7f550b257640 1 --2- 172.21.15.186:0/87916622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55040c5380 0x7f55040cae50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.538+0000 7f550b257640 1 -- 172.21.15.186:0/87916622 >> 172.21.15.186:0/87916622 conn(0x7f55040d00f0 msgr2=0x7f55040d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.538+0000 7f550b257640 1 -- 172.21.15.186:0/87916622 shutdown_connections 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.538+0000 7f550b257640 1 -- 172.21.15.186:0/87916622 wait complete. 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.539+0000 7f550b257640 1 Processor -- start 2024-03-28T15:01:32.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.539+0000 7f550b257640 1 -- start start 2024-03-28T15:01:32.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.540+0000 7f550b257640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55040c5380 0x7f55040b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.540+0000 7f550b257640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55040caa70 0x7f55041f0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.540+0000 7f550b257640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55040cb390 0x7f55041f0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.540+0000 7f550b257640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f55040c5cc0 con 0x7f55040cb390 2024-03-28T15:01:32.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.540+0000 7f550b257640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f550415b530 con 0x7f55040caa70 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.540+0000 7f550b257640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f55040cb840 con 0x7f55040c5380 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.540+0000 7f55097f4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55040cb390 0x7f55041f0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.540+0000 7f55097f4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55040cb390 0x7f55041f0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41076/0 (socket says 172.21.15.186:41076) 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.540+0000 7f55097f4640 1 -- 172.21.15.186:0/2107536808 learned_addr learned my addr 172.21.15.186:0/2107536808 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.540+0000 7f5503fff640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55040caa70 0x7f55041f0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.540+0000 7f5508ff3640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55040c5380 0x7f55040b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.541+0000 7f55097f4640 1 -- 172.21.15.186:0/2107536808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55040c5380 msgr2=0x7f55040b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.541+0000 7f55097f4640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55040c5380 0x7f55040b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.541+0000 7f55097f4640 1 -- 172.21.15.186:0/2107536808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55040caa70 msgr2=0x7f55041f0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.541+0000 7f55097f4640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55040caa70 0x7f55041f0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.541+0000 7f55097f4640 1 -- 172.21.15.186:0/2107536808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55041f3a50 con 0x7f55040cb390 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.541+0000 7f55097f4640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55040cb390 0x7f55041f0aa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00cd50 tx=0x7f54fc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.541+0000 7f55017fa640 1 -- 172.21.15.186:0/2107536808 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc00f680 con 0x7f55040cb390 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.541+0000 7f55017fa640 1 -- 172.21.15.186:0/2107536808 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54fc00f840 con 0x7f55040cb390 2024-03-28T15:01:32.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.541+0000 7f55017fa640 1 -- 172.21.15.186:0/2107536808 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc00fc30 con 0x7f55040cb390 2024-03-28T15:01:32.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.541+0000 7f550b257640 1 -- 172.21.15.186:0/2107536808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55041f3d30 con 0x7f55040cb390 2024-03-28T15:01:32.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.541+0000 7f550b257640 1 -- 172.21.15.186:0/2107536808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55041f42f0 con 0x7f55040cb390 2024-03-28T15:01:32.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.542+0000 7f550b257640 1 -- 172.21.15.186:0/2107536808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55040686d0 con 0x7f55040cb390 2024-03-28T15:01:32.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.542+0000 7f55017fa640 1 -- 172.21.15.186:0/2107536808 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f54fc00e3e0 con 0x7f55040cb390 2024-03-28T15:01:32.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.543+0000 7f55017fa640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f54e806c560 0x7f54e806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.543+0000 7f55017fa640 1 -- 172.21.15.186:0/2107536808 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f54fc08c2d0 con 0x7f55040cb390 2024-03-28T15:01:32.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.543+0000 7f5508ff3640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f54e806c560 0x7f54e806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.543+0000 7f5508ff3640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f54e806c560 0x7f54e806ea20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f54f8001780 tx=0x7f54f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:32.547 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.547+0000 7f55017fa640 1 -- 172.21.15.186:0/2107536808 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f54fc014030 con 0x7f55040cb390 2024-03-28T15:01:32.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.724+0000 7f550b257640 1 -- 172.21.15.186:0/2107536808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55040b3b10 con 0x7f55040cb390 2024-03-28T15:01:32.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.725+0000 7f55017fa640 1 -- 172.21.15.186:0/2107536808 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16654 (secure 0 0 0) 0x7f54fc05a2a0 con 0x7f55040cb390 2024-03-28T15:01:32.725 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:32.726 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:31.424983+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 -- 172.21.15.186:0/2107536808 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f54e806c560 msgr2=0x7f54e806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f54e806c560 0x7f54e806ea20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f54f8001780 tx=0x7f54f8009040 comp rx=0 tx=0).stop 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 -- 172.21.15.186:0/2107536808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55040cb390 msgr2=0x7f55041f0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55040cb390 0x7f55041f0aa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00cd50 tx=0x7f54fc00a650 comp rx=0 tx=0).stop 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 -- 172.21.15.186:0/2107536808 shutdown_connections 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f54e806c560 0x7f54e806ea20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55040cb390 0x7f55041f0aa0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55040caa70 0x7f55041f0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 --2- 172.21.15.186:0/2107536808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55040c5380 0x7f55040b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 -- 172.21.15.186:0/2107536808 >> 172.21.15.186:0/2107536808 conn(0x7f55040d00f0 msgr2=0x7f55040ccb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 -- 172.21.15.186:0/2107536808 shutdown_connections 2024-03-28T15:01:32.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.727+0000 7f550b257640 1 -- 172.21.15.186:0/2107536808 wait complete. 2024-03-28T15:01:32.738 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T15:01:32.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.842+0000 7f02d74c9640 1 Processor -- start 2024-03-28T15:01:32.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.842+0000 7f02d74c9640 1 -- start start 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d74c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02d00cb0b0 0x7f02d015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d74c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02d00cce90 0x7f02d00cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d74c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02d015de90 con 0x7f02d0154a10 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d74c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f02d00c5ae0 con 0x7f02d00cce90 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d74c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f02d015e750 con 0x7f02d00cb0b0 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d5a66640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02d00cce90 0x7f02d00cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d5a66640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02d00cce90 0x7f02d00cd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41084/0 (socket says 172.21.15.186:41084) 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d5a66640 1 -- 172.21.15.186:0/214304149 learned_addr learned my addr 172.21.15.186:0/214304149 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d4a64640 1 --2- 172.21.15.186:0/214304149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02d00cb0b0 0x7f02d015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d5a66640 1 -- 172.21.15.186:0/214304149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02d00cb0b0 msgr2=0x7f02d015b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d5a66640 1 --2- 172.21.15.186:0/214304149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02d00cb0b0 0x7f02d015b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d5a66640 1 -- 172.21.15.186:0/214304149 >> v1:172.21.15.186:6789/0 conn(0x7f02d0154a10 legacy=0x7f02d00c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.843+0000 7f02d5a66640 1 -- 172.21.15.186:0/214304149 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d00cb6d0 con 0x7f02d00cce90 2024-03-28T15:01:32.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02d5a66640 1 --2- 172.21.15.186:0/214304149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02d00cce90 0x7f02d00cd2e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f02c8009e50 tx=0x7f02c802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cfe9832897b2bac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:32.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02c7fff640 1 -- 172.21.15.186:0/214304149 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c802cb90 con 0x7f02d00cce90 2024-03-28T15:01:32.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02c7fff640 1 -- 172.21.15.186:0/214304149 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02c802cd30 con 0x7f02d00cce90 2024-03-28T15:01:32.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02c7fff640 1 -- 172.21.15.186:0/214304149 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c8034610 con 0x7f02d00cce90 2024-03-28T15:01:32.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02d74c9640 1 -- 172.21.15.186:0/214304149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02d00cce90 msgr2=0x7f02d00cd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02d74c9640 1 --2- 172.21.15.186:0/214304149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02d00cce90 0x7f02d00cd2e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f02c8009e50 tx=0x7f02c802bff0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02d74c9640 1 -- 172.21.15.186:0/214304149 shutdown_connections 2024-03-28T15:01:32.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02d74c9640 1 --2- 172.21.15.186:0/214304149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02d00cb0b0 0x7f02d015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02d74c9640 1 --2- 172.21.15.186:0/214304149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02d00cce90 0x7f02d00cd2e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02d74c9640 1 -- 172.21.15.186:0/214304149 >> 172.21.15.186:0/214304149 conn(0x7f02d006b640 msgr2=0x7f02d006ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:32.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02d74c9640 1 -- 172.21.15.186:0/214304149 shutdown_connections 2024-03-28T15:01:32.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.844+0000 7f02d74c9640 1 -- 172.21.15.186:0/214304149 wait complete. 2024-03-28T15:01:32.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.845+0000 7f02d74c9640 1 Processor -- start 2024-03-28T15:01:32.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.846+0000 7f02d74c9640 1 -- start start 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.846+0000 7f02d74c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02d00cb0b0 0x7f02d00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.846+0000 7f02d74c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02d00cce90 0x7f02d00b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.846+0000 7f02d74c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02d0154a10 0x7f02d015c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.846+0000 7f02d74c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f02d015e3e0 con 0x7f02d00cce90 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.846+0000 7f02d74c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f02d015b800 con 0x7f02d0154a10 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.846+0000 7f02d74c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f02d00cb9b0 con 0x7f02d00cb0b0 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.846+0000 7f02d4a64640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02d00cce90 0x7f02d00b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.846+0000 7f02d4a64640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02d00cce90 0x7f02d00b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41098/0 (socket says 172.21.15.186:41098) 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.846+0000 7f02d4a64640 1 -- 172.21.15.186:0/3763313725 learned_addr learned my addr 172.21.15.186:0/3763313725 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.846+0000 7f02d5265640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02d00cb0b0 0x7f02d00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02d4a64640 1 -- 172.21.15.186:0/3763313725 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02d00cb0b0 msgr2=0x7f02d00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02d4a64640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02d00cb0b0 0x7f02d00b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02d4a64640 1 -- 172.21.15.186:0/3763313725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02d0154a10 msgr2=0x7f02d015c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:32.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02d4a64640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02d0154a10 0x7f02d015c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:32.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02d4a64640 1 -- 172.21.15.186:0/3763313725 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d00b5c40 con 0x7f02d00cce90 2024-03-28T15:01:32.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02d5265640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02d00cb0b0 0x7f02d00b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:32.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02d4a64640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02d00cce90 0x7f02d00b42f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f02c000d800 tx=0x7f02c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:32.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02c5ffb640 1 -- 172.21.15.186:0/3763313725 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c000b660 con 0x7f02d00cce90 2024-03-28T15:01:32.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02c5ffb640 1 -- 172.21.15.186:0/3763313725 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02c0019070 con 0x7f02d00cce90 2024-03-28T15:01:32.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02c5ffb640 1 -- 172.21.15.186:0/3763313725 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c000ba20 con 0x7f02d00cce90 2024-03-28T15:01:32.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02d74c9640 1 -- 172.21.15.186:0/3763313725 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02d00b5ed0 con 0x7f02d00cce90 2024-03-28T15:01:32.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.847+0000 7f02d74c9640 1 -- 172.21.15.186:0/3763313725 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02d01f0930 con 0x7f02d00cce90 2024-03-28T15:01:32.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.848+0000 7f02d74c9640 1 -- 172.21.15.186:0/3763313725 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02d0069570 con 0x7f02d00cce90 2024-03-28T15:01:32.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.848+0000 7f02c5ffb640 1 -- 172.21.15.186:0/3763313725 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f02c000bbc0 con 0x7f02d00cce90 2024-03-28T15:01:32.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.849+0000 7f02c5ffb640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f02b406c530 0x7f02b406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:32.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.849+0000 7f02c5ffb640 1 -- 172.21.15.186:0/3763313725 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f02c008bba0 con 0x7f02d00cce90 2024-03-28T15:01:32.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.849+0000 7f02d5265640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f02b406c530 0x7f02b406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:32.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.849+0000 7f02d5265640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f02b406c530 0x7f02b406e9f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0015e0 tx=0x7f02cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:32.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:32.853+0000 7f02c5ffb640 1 -- 172.21.15.186:0/3763313725 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f02c001c360 con 0x7f02d00cce90 2024-03-28T15:01:33.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.070+0000 7f02d74c9640 1 -- 172.21.15.186:0/3763313725 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f02d014ea90 con 0x7f02d00cce90 2024-03-28T15:01:33.114 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.114+0000 7f02c5ffb640 1 -- 172.21.15.186:0/3763313725 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7f02c005a1c0 con 0x7f02d00cce90 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.116+0000 7f02d74c9640 1 -- 172.21.15.186:0/3763313725 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f02b406c530 msgr2=0x7f02b406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.116+0000 7f02d74c9640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f02b406c530 0x7f02b406e9f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0015e0 tx=0x7f02cc009040 comp rx=0 tx=0).stop 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.116+0000 7f02d74c9640 1 -- 172.21.15.186:0/3763313725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02d00cce90 msgr2=0x7f02d00b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.116+0000 7f02d74c9640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02d00cce90 0x7f02d00b42f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f02c000d800 tx=0x7f02c000dcc0 comp rx=0 tx=0).stop 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.116+0000 7f02d74c9640 1 -- 172.21.15.186:0/3763313725 shutdown_connections 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.116+0000 7f02d74c9640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f02b406c530 0x7f02b406e9f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.116+0000 7f02d74c9640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02d0154a10 0x7f02d015c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.116+0000 7f02d74c9640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02d00cce90 0x7f02d00b42f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.116+0000 7f02d74c9640 1 --2- 172.21.15.186:0/3763313725 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02d00cb0b0 0x7f02d00b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.116+0000 7f02d74c9640 1 -- 172.21.15.186:0/3763313725 >> 172.21.15.186:0/3763313725 conn(0x7f02d006b640 msgr2=0x7f02d0152fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.116+0000 7f02d74c9640 1 -- 172.21.15.186:0/3763313725 shutdown_connections 2024-03-28T15:01:33.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.117+0000 7f02d74c9640 1 -- 172.21.15.186:0/3763313725 wait complete. 2024-03-28T15:01:33.127 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-03-28T15:01:33.127 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T15:01:33.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.230+0000 7f875f888640 1 Processor -- start 2024-03-28T15:01:33.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.230+0000 7f875f888640 1 -- start start 2024-03-28T15:01:33.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.231+0000 7f875f888640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f87580719a0 0x7f87580ca380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:33.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.231+0000 7f875f888640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87580c8a20 0x7f875806e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:33.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.231+0000 7f875f888640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87580cae60 con 0x7f87580ca8c0 2024-03-28T15:01:33.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.231+0000 7f875f888640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8758070e90 con 0x7f87580719a0 2024-03-28T15:01:33.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.231+0000 7f875f888640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f875806b460 con 0x7f87580c8a20 2024-03-28T15:01:33.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.231+0000 7f875d624640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f87580719a0 0x7f87580ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:33.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.231+0000 7f875d624640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f87580719a0 0x7f87580ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41114/0 (socket says 172.21.15.186:41114) 2024-03-28T15:01:33.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.231+0000 7f875d624640 1 -- 172.21.15.186:0/3194066630 learned_addr learned my addr 172.21.15.186:0/3194066630 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:33.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.231+0000 7f875ce23640 1 --2- 172.21.15.186:0/3194066630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87580c8a20 0x7f875806e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:33.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.232+0000 7f875d624640 1 -- 172.21.15.186:0/3194066630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87580c8a20 msgr2=0x7f875806e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:33.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.232+0000 7f875d624640 1 --2- 172.21.15.186:0/3194066630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87580c8a20 0x7f875806e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:33.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.232+0000 7f875d624640 1 -- 172.21.15.186:0/3194066630 >> v1:172.21.15.191:6789/0 conn(0x7f87580ca8c0 legacy=0x7f875815a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:33.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.232+0000 7f875d624640 1 -- 172.21.15.186:0/3194066630 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87580cb140 con 0x7f87580719a0 2024-03-28T15:01:33.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.232+0000 7f875ce23640 1 --2- 172.21.15.186:0/3194066630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87580c8a20 0x7f875806e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:33.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.232+0000 7f875d624640 1 --2- 172.21.15.186:0/3194066630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87580719a0 0x7f87580ca380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8754009e50 tx=0x7f875402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617d7aa7e40ad729 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.232+0000 7f874ffff640 1 -- 172.21.15.186:0/3194066630 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875402cb00 con 0x7f87580719a0 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.233+0000 7f874ffff640 1 -- 172.21.15.186:0/3194066630 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875402cca0 con 0x7f87580719a0 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.233+0000 7f874ffff640 1 -- 172.21.15.186:0/3194066630 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8754034610 con 0x7f87580719a0 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.233+0000 7f875f888640 1 -- 172.21.15.186:0/3194066630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87580719a0 msgr2=0x7f87580ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.233+0000 7f875f888640 1 --2- 172.21.15.186:0/3194066630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87580719a0 0x7f87580ca380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8754009e50 tx=0x7f875402bf60 comp rx=0 tx=0).stop 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.233+0000 7f875f888640 1 -- 172.21.15.186:0/3194066630 shutdown_connections 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.233+0000 7f875f888640 1 --2- 172.21.15.186:0/3194066630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87580c8a20 0x7f875806e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.233+0000 7f875f888640 1 --2- 172.21.15.186:0/3194066630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87580719a0 0x7f87580ca380 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.233+0000 7f875f888640 1 -- 172.21.15.186:0/3194066630 >> 172.21.15.186:0/3194066630 conn(0x7f875806fa00 msgr2=0x7f8758073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.233+0000 7f875f888640 1 -- 172.21.15.186:0/3194066630 shutdown_connections 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.233+0000 7f875f888640 1 -- 172.21.15.186:0/3194066630 wait complete. 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.234+0000 7f875f888640 1 Processor -- start 2024-03-28T15:01:33.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.234+0000 7f875f888640 1 -- start start 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875f888640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87580719a0 0x7f8758154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875f888640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87580c8a20 0x7f875814e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875f888640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87580ca8c0 0x7f87581559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875f888640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f875806b990 con 0x7f87580719a0 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875f888640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8758071d80 con 0x7f87580ca8c0 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875f888640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f875806bb10 con 0x7f87580c8a20 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875d624640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87580719a0 0x7f8758154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875d624640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87580719a0 0x7f8758154660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41130/0 (socket says 172.21.15.186:41130) 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875d624640 1 -- 172.21.15.186:0/1732456699 learned_addr learned my addr 172.21.15.186:0/1732456699 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875ce23640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87580c8a20 0x7f875814e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875de25640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87580ca8c0 0x7f87581559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875d624640 1 -- 172.21.15.186:0/1732456699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87580c8a20 msgr2=0x7f875814e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:33.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875d624640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87580c8a20 0x7f875814e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875d624640 1 -- 172.21.15.186:0/1732456699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87580ca8c0 msgr2=0x7f87581559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875d624640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87580ca8c0 0x7f87581559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875d624640 1 -- 172.21.15.186:0/1732456699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8758154ba0 con 0x7f87580719a0 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875de25640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87580ca8c0 0x7f87581559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.235+0000 7f875d624640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87580719a0 0x7f8758154660 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8754036040 tx=0x7f8754004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.236+0000 7f874dffb640 1 -- 172.21.15.186:0/1732456699 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8754043020 con 0x7f87580719a0 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.236+0000 7f874dffb640 1 -- 172.21.15.186:0/1732456699 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875402ce50 con 0x7f87580719a0 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.236+0000 7f875f888640 1 -- 172.21.15.186:0/1732456699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8758154e10 con 0x7f87580719a0 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.236+0000 7f874dffb640 1 -- 172.21.15.186:0/1732456699 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8754009510 con 0x7f87580719a0 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.236+0000 7f875f888640 1 -- 172.21.15.186:0/1732456699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87581567a0 con 0x7f87580719a0 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.237+0000 7f875f888640 1 -- 172.21.15.186:0/1732456699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87580686d0 con 0x7f87580719a0 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.237+0000 7f874dffb640 1 -- 172.21.15.186:0/1732456699 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8754046020 con 0x7f87580719a0 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.238+0000 7f874dffb640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f873c06c530 0x7f873c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.238+0000 7f874dffb640 1 -- 172.21.15.186:0/1732456699 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f87540addf0 con 0x7f87580719a0 2024-03-28T15:01:33.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.238+0000 7f875ce23640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f873c06c530 0x7f873c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:33.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.238+0000 7f875ce23640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f873c06c530 0x7f873c06e9f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f87580ca480 tx=0x7f8748008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:33.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.242+0000 7f874dffb640 1 -- 172.21.15.186:0/1732456699 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f875404d030 con 0x7f87580719a0 2024-03-28T15:01:33.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:33.456+0000 7f875f888640 1 -- 172.21.15.186:0/1732456699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f87580c4ca0 con 0x7f87580719a0 2024-03-28T15:01:34.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.115+0000 7f874dffb640 1 -- 172.21.15.186:0/1732456699 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) ==== 122+0+0 (secure 0 0 0) 0x7f875407c410 con 0x7f87580719a0 2024-03-28T15:01:34.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.117+0000 7f875f888640 1 -- 172.21.15.186:0/1732456699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f873c06c530 msgr2=0x7f873c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:34.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.117+0000 7f875f888640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f873c06c530 0x7f873c06e9f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f87580ca480 tx=0x7f8748008040 comp rx=0 tx=0).stop 2024-03-28T15:01:34.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.117+0000 7f875f888640 1 -- 172.21.15.186:0/1732456699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87580719a0 msgr2=0x7f8758154660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:34.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.117+0000 7f875f888640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87580719a0 0x7f8758154660 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8754036040 tx=0x7f8754004a40 comp rx=0 tx=0).stop 2024-03-28T15:01:34.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.118+0000 7f875f888640 1 -- 172.21.15.186:0/1732456699 shutdown_connections 2024-03-28T15:01:34.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.118+0000 7f875f888640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f873c06c530 0x7f873c06e9f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:34.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.118+0000 7f875f888640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87580ca8c0 0x7f87581559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:34.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.118+0000 7f875f888640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87580c8a20 0x7f875814e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:34.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.118+0000 7f875f888640 1 --2- 172.21.15.186:0/1732456699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87580719a0 0x7f8758154660 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:34.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.118+0000 7f875f888640 1 -- 172.21.15.186:0/1732456699 >> 172.21.15.186:0/1732456699 conn(0x7f875806fa00 msgr2=0x7f8758073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:34.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.118+0000 7f875f888640 1 -- 172.21.15.186:0/1732456699 shutdown_connections 2024-03-28T15:01:34.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.118+0000 7f875f888640 1 -- 172.21.15.186:0/1732456699 wait complete. 2024-03-28T15:01:34.129 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T15:01:34.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.234+0000 7efe737b6640 1 Processor -- start 2024-03-28T15:01:34.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.234+0000 7efe737b6640 1 -- start start 2024-03-28T15:01:34.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe737b6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efe6c154af0 0x7efe6c0c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:34.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe737b6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efe6c0cc2b0 0x7efe6c15de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe737b6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe6c15a950 con 0x7efe6c072780 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe737b6640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7efe6c15abc0 con 0x7efe6c154af0 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe737b6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7efe6c15aea0 con 0x7efe6c0cc2b0 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe71552640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efe6c154af0 0x7efe6c0c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe71552640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efe6c154af0 0x7efe6c0c5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41136/0 (socket says 172.21.15.186:41136) 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe71552640 1 -- 172.21.15.186:0/2511671434 learned_addr learned my addr 172.21.15.186:0/2511671434 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe71d53640 1 --2- 172.21.15.186:0/2511671434 >> v2:172.21.15.191:3300/0 conn(0x7efe6c0cc2b0 0x7efe6c15de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe71552640 1 -- 172.21.15.186:0/2511671434 >> v2:172.21.15.191:3300/0 conn(0x7efe6c0cc2b0 msgr2=0x7efe6c15de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe71552640 1 --2- 172.21.15.186:0/2511671434 >> v2:172.21.15.191:3300/0 conn(0x7efe6c0cc2b0 0x7efe6c15de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe71552640 1 -- 172.21.15.186:0/2511671434 >> v1:172.21.15.186:6789/0 conn(0x7efe6c072780 legacy=0x7efe6c0d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe71552640 1 -- 172.21.15.186:0/2511671434 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6c15b180 con 0x7efe6c154af0 2024-03-28T15:01:34.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.235+0000 7efe71d53640 1 --2- 172.21.15.186:0/2511671434 >> v2:172.21.15.191:3300/0 conn(0x7efe6c0cc2b0 0x7efe6c15de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:34.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.236+0000 7efe71552640 1 --2- 172.21.15.186:0/2511671434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe6c154af0 0x7efe6c0c5da0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efe6c072670 tx=0x7efe6802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15551e3ab9134597 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:34.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.236+0000 7efe63fff640 1 -- 172.21.15.186:0/2511671434 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6802f850 con 0x7efe6c154af0 2024-03-28T15:01:34.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.236+0000 7efe63fff640 1 -- 172.21.15.186:0/2511671434 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe6802f9f0 con 0x7efe6c154af0 2024-03-28T15:01:34.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.236+0000 7efe63fff640 1 -- 172.21.15.186:0/2511671434 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6802fdc0 con 0x7efe6c154af0 2024-03-28T15:01:34.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.237+0000 7efe737b6640 1 -- 172.21.15.186:0/2511671434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe6c154af0 msgr2=0x7efe6c0c5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:34.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.237+0000 7efe737b6640 1 --2- 172.21.15.186:0/2511671434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe6c154af0 0x7efe6c0c5da0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efe6c072670 tx=0x7efe6802ba30 comp rx=0 tx=0).stop 2024-03-28T15:01:34.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.237+0000 7efe737b6640 1 -- 172.21.15.186:0/2511671434 shutdown_connections 2024-03-28T15:01:34.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.237+0000 7efe737b6640 1 --2- 172.21.15.186:0/2511671434 >> v2:172.21.15.191:3300/0 conn(0x7efe6c0cc2b0 0x7efe6c15de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:34.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.237+0000 7efe737b6640 1 --2- 172.21.15.186:0/2511671434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe6c154af0 0x7efe6c0c5da0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:34.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.237+0000 7efe737b6640 1 -- 172.21.15.186:0/2511671434 >> 172.21.15.186:0/2511671434 conn(0x7efe6c073580 msgr2=0x7efe6c073990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:34.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.237+0000 7efe737b6640 1 -- 172.21.15.186:0/2511671434 shutdown_connections 2024-03-28T15:01:34.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.237+0000 7efe737b6640 1 -- 172.21.15.186:0/2511671434 wait complete. 2024-03-28T15:01:34.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.238+0000 7efe737b6640 1 Processor -- start 2024-03-28T15:01:34.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.238+0000 7efe737b6640 1 -- start start 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe737b6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe6c072780 0x7efe6c15dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe737b6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe6c0cc2b0 0x7efe6c1f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe737b6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe6c154af0 0x7efe6c1f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe737b6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c154ed0 con 0x7efe6c154af0 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe737b6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c0c62e0 con 0x7efe6c0cc2b0 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe737b6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7efe6c0cd800 con 0x7efe6c072780 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe71d53640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe6c154af0 0x7efe6c1f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe71d53640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe6c154af0 0x7efe6c1f0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41148/0 (socket says 172.21.15.186:41148) 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe71d53640 1 -- 172.21.15.186:0/1865766366 learned_addr learned my addr 172.21.15.186:0/1865766366 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe70d51640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe6c0cc2b0 0x7efe6c1f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe71d53640 1 -- 172.21.15.186:0/1865766366 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe6c072780 msgr2=0x7efe6c15dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:34.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe71d53640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe6c072780 0x7efe6c15dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:34.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe71d53640 1 -- 172.21.15.186:0/1865766366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe6c0cc2b0 msgr2=0x7efe6c1f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:34.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe71d53640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe6c0cc2b0 0x7efe6c1f4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:34.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe71d53640 1 -- 172.21.15.186:0/1865766366 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6c1f1240 con 0x7efe6c154af0 2024-03-28T15:01:34.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe71d53640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe6c154af0 0x7efe6c1f0d00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efe6400cf20 tx=0x7efe6400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:34.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe61ffb640 1 -- 172.21.15.186:0/1865766366 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6401a070 con 0x7efe6c154af0 2024-03-28T15:01:34.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe61ffb640 1 -- 172.21.15.186:0/1865766366 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe6400d840 con 0x7efe6c154af0 2024-03-28T15:01:34.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe61ffb640 1 -- 172.21.15.186:0/1865766366 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6400dc30 con 0x7efe6c154af0 2024-03-28T15:01:34.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.239+0000 7efe737b6640 1 -- 172.21.15.186:0/1865766366 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe6c15e700 con 0x7efe6c154af0 2024-03-28T15:01:34.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.240+0000 7efe737b6640 1 -- 172.21.15.186:0/1865766366 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe6c1f0500 con 0x7efe6c154af0 2024-03-28T15:01:34.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.241+0000 7efe737b6640 1 -- 172.21.15.186:0/1865766366 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe6c1f02f0 con 0x7efe6c154af0 2024-03-28T15:01:34.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.241+0000 7efe61ffb640 1 -- 172.21.15.186:0/1865766366 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7efe6401d070 con 0x7efe6c154af0 2024-03-28T15:01:34.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.241+0000 7efe61ffb640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7efe5006c530 0x7efe5006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:34.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.241+0000 7efe61ffb640 1 -- 172.21.15.186:0/1865766366 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efe6408bf40 con 0x7efe6c154af0 2024-03-28T15:01:34.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.241+0000 7efe71552640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7efe5006c530 0x7efe5006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:34.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.242+0000 7efe71552640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7efe5006c530 0x7efe5006e9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efe68036000 tx=0x7efe68039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:34.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.246+0000 7efe61ffb640 1 -- 172.21.15.186:0/1865766366 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7efe6405a5f0 con 0x7efe6c154af0 2024-03-28T15:01:34.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:34.457+0000 7efe737b6640 1 -- 172.21.15.186:0/1865766366 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7efe6c069570 con 0x7efe6c154af0 2024-03-28T15:01:35.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.119+0000 7efe61ffb640 1 -- 172.21.15.186:0/1865766366 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v7) ==== 115+0+0 (secure 0 0 0) 0x7efe64059ef0 con 0x7efe6c154af0 2024-03-28T15:01:35.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 -- 172.21.15.186:0/1865766366 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7efe5006c530 msgr2=0x7efe5006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7efe5006c530 0x7efe5006e9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efe68036000 tx=0x7efe68039040 comp rx=0 tx=0).stop 2024-03-28T15:01:35.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 -- 172.21.15.186:0/1865766366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe6c154af0 msgr2=0x7efe6c1f0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe6c154af0 0x7efe6c1f0d00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efe6400cf20 tx=0x7efe6400a7d0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 -- 172.21.15.186:0/1865766366 shutdown_connections 2024-03-28T15:01:35.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7efe5006c530 0x7efe5006e9f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe6c154af0 0x7efe6c1f0d00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe6c0cc2b0 0x7efe6c1f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 --2- 172.21.15.186:0/1865766366 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe6c072780 0x7efe6c15dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 -- 172.21.15.186:0/1865766366 >> 172.21.15.186:0/1865766366 conn(0x7efe6c073580 msgr2=0x7efe6c0cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:35.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 -- 172.21.15.186:0/1865766366 shutdown_connections 2024-03-28T15:01:35.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.122+0000 7efe737b6640 1 -- 172.21.15.186:0/1865766366 wait complete. 2024-03-28T15:01:35.134 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T15:01:35.134 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:01:35.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.235+0000 7f849049a640 1 Processor -- start 2024-03-28T15:01:35.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.235+0000 7f849049a640 1 -- start start 2024-03-28T15:01:35.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.236+0000 7f849049a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f848806d670 0x7f848806f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.236+0000 7f849049a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848806fb90 0x7f84880c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.236+0000 7f849049a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848806f770 con 0x7f848806da50 2024-03-28T15:01:35.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.236+0000 7f849049a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f84880c4c60 con 0x7f848806d670 2024-03-28T15:01:35.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.236+0000 7f849049a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8488159500 con 0x7f848806fb90 2024-03-28T15:01:35.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.236+0000 7f848e236640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f848806d670 0x7f848806f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.237+0000 7f848e236640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f848806d670 0x7f848806f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41154/0 (socket says 172.21.15.186:41154) 2024-03-28T15:01:35.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.237+0000 7f848e236640 1 -- 172.21.15.186:0/2485189693 learned_addr learned my addr 172.21.15.186:0/2485189693 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:35.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.237+0000 7f848da35640 1 --2- 172.21.15.186:0/2485189693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848806fb90 0x7f84880c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.237+0000 7f848e236640 1 -- 172.21.15.186:0/2485189693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848806fb90 msgr2=0x7f84880c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:35.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.237+0000 7f848e236640 1 --2- 172.21.15.186:0/2485189693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848806fb90 0x7f84880c8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.237+0000 7f848e236640 1 -- 172.21.15.186:0/2485189693 >> v1:172.21.15.186:6789/0 conn(0x7f848806da50 legacy=0x7f848815bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.237+0000 7f848e236640 1 -- 172.21.15.186:0/2485189693 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8488159220 con 0x7f848806d670 2024-03-28T15:01:35.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.237+0000 7f848e236640 1 --2- 172.21.15.186:0/2485189693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f848806d670 0x7f848806f0a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f847c009870 tx=0x7f847c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dffa479150fb9acb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:35.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.238+0000 7f848d234640 1 -- 172.21.15.186:0/2485189693 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847c035020 con 0x7f848806d670 2024-03-28T15:01:35.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.238+0000 7f848d234640 1 -- 172.21.15.186:0/2485189693 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f847c031640 con 0x7f848806d670 2024-03-28T15:01:35.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.238+0000 7f848d234640 1 -- 172.21.15.186:0/2485189693 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847c02d040 con 0x7f848806d670 2024-03-28T15:01:35.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.238+0000 7f849049a640 1 -- 172.21.15.186:0/2485189693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f848806d670 msgr2=0x7f848806f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.238+0000 7f849049a640 1 --2- 172.21.15.186:0/2485189693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f848806d670 0x7f848806f0a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f847c009870 tx=0x7f847c02b8f0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.238+0000 7f849049a640 1 -- 172.21.15.186:0/2485189693 shutdown_connections 2024-03-28T15:01:35.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.238+0000 7f849049a640 1 --2- 172.21.15.186:0/2485189693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848806fb90 0x7f84880c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.238+0000 7f849049a640 1 --2- 172.21.15.186:0/2485189693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f848806d670 0x7f848806f0a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.238+0000 7f849049a640 1 -- 172.21.15.186:0/2485189693 >> 172.21.15.186:0/2485189693 conn(0x7f8488067670 msgr2=0x7f8488071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:35.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.238+0000 7f849049a640 1 -- 172.21.15.186:0/2485189693 shutdown_connections 2024-03-28T15:01:35.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.238+0000 7f849049a640 1 -- 172.21.15.186:0/2485189693 wait complete. 2024-03-28T15:01:35.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.239+0000 7f849049a640 1 Processor -- start 2024-03-28T15:01:35.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.239+0000 7f849049a640 1 -- start start 2024-03-28T15:01:35.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.240+0000 7f849049a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848806d670 0x7f84881ee140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.240+0000 7f849049a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f848806da50 0x7f8488151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.240+0000 7f849049a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f848806fb90 0x7f8488144bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.240+0000 7f849049a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f84880c50f0 con 0x7f848806fb90 2024-03-28T15:01:35.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.240+0000 7f849049a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f84880c34c0 con 0x7f848806da50 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.240+0000 7f849049a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f84880c56b0 con 0x7f848806d670 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.240+0000 7f848ea37640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f848806fb90 0x7f8488144bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.240+0000 7f848ea37640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f848806fb90 0x7f8488144bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41156/0 (socket says 172.21.15.186:41156) 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.240+0000 7f848ea37640 1 -- 172.21.15.186:0/823306945 learned_addr learned my addr 172.21.15.186:0/823306945 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.240+0000 7f848e236640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848806d670 0x7f84881ee140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.240+0000 7f848da35640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f848806da50 0x7f8488151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f848ea37640 1 -- 172.21.15.186:0/823306945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848806d670 msgr2=0x7f84881ee140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f848ea37640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848806d670 0x7f84881ee140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f848ea37640 1 -- 172.21.15.186:0/823306945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f848806da50 msgr2=0x7f8488151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f848ea37640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f848806da50 0x7f8488151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f848ea37640 1 -- 172.21.15.186:0/823306945 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84880c7000 con 0x7f848806fb90 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f848da35640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f848806da50 0x7f8488151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f848ea37640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f848806fb90 0x7f8488144bf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f848000c6e0 tx=0x7f848000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f847affd640 1 -- 172.21.15.186:0/823306945 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8480003b70 con 0x7f848806fb90 2024-03-28T15:01:35.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f847affd640 1 -- 172.21.15.186:0/823306945 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8480003d30 con 0x7f848806fb90 2024-03-28T15:01:35.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f849049a640 1 -- 172.21.15.186:0/823306945 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84880c72e0 con 0x7f848806fb90 2024-03-28T15:01:35.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f847affd640 1 -- 172.21.15.186:0/823306945 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8480005310 con 0x7f848806fb90 2024-03-28T15:01:35.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.241+0000 7f849049a640 1 -- 172.21.15.186:0/823306945 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84880c5c30 con 0x7f848806fb90 2024-03-28T15:01:35.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.242+0000 7f849049a640 1 -- 172.21.15.186:0/823306945 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84880686d0 con 0x7f848806fb90 2024-03-28T15:01:35.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.242+0000 7f847affd640 1 -- 172.21.15.186:0/823306945 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f84800054b0 con 0x7f848806fb90 2024-03-28T15:01:35.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.243+0000 7f847affd640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f846c06c530 0x7f846c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.243+0000 7f847affd640 1 -- 172.21.15.186:0/823306945 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f848008b4a0 con 0x7f848806fb90 2024-03-28T15:01:35.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.243+0000 7f848e236640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f846c06c530 0x7f846c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.243+0000 7f848e236640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f846c06c530 0x7f846c06e9f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f847c036000 tx=0x7f847c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:35.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.247+0000 7f847affd640 1 -- 172.21.15.186:0/823306945 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8480059ac0 con 0x7f848806fb90 2024-03-28T15:01:35.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.462+0000 7f849049a640 1 -- 172.21.15.186:0/823306945 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f848806f970 con 0x7f848806fb90 2024-03-28T15:01:35.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.462+0000 7f847affd640 1 -- 172.21.15.186:0/823306945 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6834 (secure 0 0 0) 0x7f84800593c0 con 0x7f848806fb90 2024-03-28T15:01:35.463 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:35.463 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:34.458921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:01:35.463 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 7 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 -- 172.21.15.186:0/823306945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f846c06c530 msgr2=0x7f846c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f846c06c530 0x7f846c06e9f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f847c036000 tx=0x7f847c03c040 comp rx=0 tx=0).stop 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 -- 172.21.15.186:0/823306945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f848806fb90 msgr2=0x7f8488144bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f848806fb90 0x7f8488144bf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f848000c6e0 tx=0x7f848000cba0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 -- 172.21.15.186:0/823306945 shutdown_connections 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f846c06c530 0x7f846c06e9f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f848806fb90 0x7f8488144bf0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f848806da50 0x7f8488151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 --2- 172.21.15.186:0/823306945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848806d670 0x7f84881ee140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 -- 172.21.15.186:0/823306945 >> 172.21.15.186:0/823306945 conn(0x7f8488067670 msgr2=0x7f8488151d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 -- 172.21.15.186:0/823306945 shutdown_connections 2024-03-28T15:01:35.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.464+0000 7f849049a640 1 -- 172.21.15.186:0/823306945 wait complete. 2024-03-28T15:01:35.475 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:35.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.575+0000 7f7dda93e640 1 Processor -- start 2024-03-28T15:01:35.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.575+0000 7f7dda93e640 1 -- start start 2024-03-28T15:01:35.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dda93e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7dd40c7060 0x7f7dd40c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dda93e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7dd40c6390 0x7f7dd40c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dda93e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dd40c8a10 con 0x7f7dd4072490 2024-03-28T15:01:35.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dda93e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f7dd40c85e0 con 0x7f7dd40c7060 2024-03-28T15:01:35.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dda93e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd415df70 con 0x7f7dd40c6390 2024-03-28T15:01:35.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dd8edb640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f7dd4072490 0x7f7dd406f2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:37398/0 (socket says 172.21.15.186:37398) 2024-03-28T15:01:35.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dd8edb640 1 -- 172.21.15.186:0/3833230311 learned_addr learned my addr 172.21.15.186:0/3833230311 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:35.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dd37fe640 1 --2- 172.21.15.186:0/3833230311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7dd40c6390 0x7f7dd40c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dd3fff640 1 --2- 172.21.15.186:0/3833230311 >> v2:172.21.15.191:3300/0 conn(0x7f7dd40c7060 0x7f7dd40c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dd2ffd640 1 -- 172.21.15.186:0/3833230311 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686232425 0 0) 0x7f7dd40c8a10 con 0x7f7dd4072490 2024-03-28T15:01:35.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dd2ffd640 1 -- 172.21.15.186:0/3833230311 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7dbc003520 con 0x7f7dd4072490 2024-03-28T15:01:35.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dd2ffd640 1 -- 172.21.15.186:0/3833230311 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3759237924 0 0) 0x7f7dbc003520 con 0x7f7dd4072490 2024-03-28T15:01:35.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dd37fe640 1 -- 172.21.15.186:0/3833230311 >> v2:172.21.15.191:3300/0 conn(0x7f7dd40c7060 msgr2=0x7f7dd40c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:35.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dd37fe640 1 --2- 172.21.15.186:0/3833230311 >> v2:172.21.15.191:3300/0 conn(0x7f7dd40c7060 0x7f7dd40c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dd37fe640 1 -- 172.21.15.186:0/3833230311 >> v1:172.21.15.186:6789/0 conn(0x7f7dd4072490 legacy=0x7f7dd406f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.576+0000 7f7dd37fe640 1 -- 172.21.15.186:0/3833230311 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd40c8cf0 con 0x7f7dd40c6390 2024-03-28T15:01:35.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.577+0000 7f7dd37fe640 1 --2- 172.21.15.186:0/3833230311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7dd40c6390 0x7f7dd40c80a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80097a0 tx=0x7f7dc802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c809a751436c2e17 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.577+0000 7f7dd2ffd640 1 -- 172.21.15.186:0/3833230311 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8037070 con 0x7f7dd40c6390 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.577+0000 7f7dd2ffd640 1 -- 172.21.15.186:0/3833230311 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc8030510 con 0x7f7dd40c6390 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.577+0000 7f7dd2ffd640 1 -- 172.21.15.186:0/3833230311 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc0019f0 con 0x7f7dd40c6390 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.578+0000 7f7dda93e640 1 -- 172.21.15.186:0/3833230311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7dd40c6390 msgr2=0x7f7dd40c80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.578+0000 7f7dda93e640 1 --2- 172.21.15.186:0/3833230311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7dd40c6390 0x7f7dd40c80a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80097a0 tx=0x7f7dc802b790 comp rx=0 tx=0).stop 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.578+0000 7f7dda93e640 1 -- 172.21.15.186:0/3833230311 shutdown_connections 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.578+0000 7f7dda93e640 1 --2- 172.21.15.186:0/3833230311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7dd40c6390 0x7f7dd40c80a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.578+0000 7f7dda93e640 1 --2- 172.21.15.186:0/3833230311 >> v2:172.21.15.191:3300/0 conn(0x7f7dd40c7060 0x7f7dd40c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.578+0000 7f7dda93e640 1 -- 172.21.15.186:0/3833230311 >> 172.21.15.186:0/3833230311 conn(0x7f7dd406b460 msgr2=0x7f7dd406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.578+0000 7f7dda93e640 1 -- 172.21.15.186:0/3833230311 shutdown_connections 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.578+0000 7f7dda93e640 1 -- 172.21.15.186:0/3833230311 wait complete. 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.579+0000 7f7dda93e640 1 Processor -- start 2024-03-28T15:01:35.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.579+0000 7f7dda93e640 1 -- start start 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dda93e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7dd4072490 0x7f7dd41f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dda93e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd40c6390 0x7f7dd40c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dda93e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dd40c7060 0x7f7dd41503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dda93e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd415e4a0 con 0x7f7dd40c6390 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dda93e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd415e810 con 0x7f7dd40c7060 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dda93e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd415eca0 con 0x7f7dd4072490 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dd37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd40c6390 0x7f7dd40c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dd37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd40c6390 0x7f7dd40c44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41170/0 (socket says 172.21.15.186:41170) 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dd37fe640 1 -- 172.21.15.186:0/2364889028 learned_addr learned my addr 172.21.15.186:0/2364889028 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dd3fff640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7dd4072490 0x7f7dd41f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dd37fe640 1 -- 172.21.15.186:0/2364889028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7dd4072490 msgr2=0x7f7dd41f3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dd8edb640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dd40c7060 0x7f7dd41503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dd37fe640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7dd4072490 0x7f7dd41f3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dd37fe640 1 -- 172.21.15.186:0/2364889028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dd40c7060 msgr2=0x7f7dd41503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dd37fe640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dd40c7060 0x7f7dd41503c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dd37fe640 1 -- 172.21.15.186:0/2364889028 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd414c900 con 0x7f7dd40c6390 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.580+0000 7f7dd37fe640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd40c6390 0x7f7dd40c44c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7dc802b7d0 tx=0x7f7dc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.581+0000 7f7dd0ff9640 1 -- 172.21.15.186:0/2364889028 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8041020 con 0x7f7dd40c6390 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.581+0000 7f7dd0ff9640 1 -- 172.21.15.186:0/2364889028 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc803c870 con 0x7f7dd40c6390 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.581+0000 7f7dda93e640 1 -- 172.21.15.186:0/2364889028 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd414c060 con 0x7f7dd40c6390 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.581+0000 7f7dd0ff9640 1 -- 172.21.15.186:0/2364889028 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8037040 con 0x7f7dd40c6390 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.581+0000 7f7dda93e640 1 -- 172.21.15.186:0/2364889028 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd414c620 con 0x7f7dd40c6390 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.582+0000 7f7dda93e640 1 -- 172.21.15.186:0/2364889028 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dd40686d0 con 0x7f7dd40c6390 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.582+0000 7f7dd0ff9640 1 -- 172.21.15.186:0/2364889028 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7dc8044020 con 0x7f7dd40c6390 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.583+0000 7f7dd0ff9640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7dc006c530 0x7f7dc006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.583+0000 7f7dd3fff640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7dc006c530 0x7f7dc006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.583+0000 7f7dd0ff9640 1 -- 172.21.15.186:0/2364889028 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7dc80ad140 con 0x7f7dd40c6390 2024-03-28T15:01:35.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.583+0000 7f7dd3fff640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7dc006c530 0x7f7dc006e9f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4009970 tx=0x7f7dc4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:35.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.587+0000 7f7dd0ff9640 1 -- 172.21.15.186:0/2364889028 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7dc804b030 con 0x7f7dd40c6390 2024-03-28T15:01:35.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.765+0000 7f7dda93e640 1 -- 172.21.15.186:0/2364889028 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7dd4062790 con 0x7f7dd40c6390 2024-03-28T15:01:35.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.766+0000 7f7dd0ff9640 1 -- 172.21.15.186:0/2364889028 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f7dc807b760 con 0x7f7dd40c6390 2024-03-28T15:01:35.767 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:35.767 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:33.099254+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 -- 172.21.15.186:0/2364889028 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7dc006c530 msgr2=0x7f7dc006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7dc006c530 0x7f7dc006e9f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4009970 tx=0x7f7dc4008040 comp rx=0 tx=0).stop 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 -- 172.21.15.186:0/2364889028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd40c6390 msgr2=0x7f7dd40c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd40c6390 0x7f7dd40c44c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7dc802b7d0 tx=0x7f7dc80046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 -- 172.21.15.186:0/2364889028 shutdown_connections 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7dc006c530 0x7f7dc006e9f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dd40c7060 0x7f7dd41503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd40c6390 0x7f7dd40c44c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 --2- 172.21.15.186:0/2364889028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7dd4072490 0x7f7dd41f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 -- 172.21.15.186:0/2364889028 >> 172.21.15.186:0/2364889028 conn(0x7f7dd406b460 msgr2=0x7f7dd406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 -- 172.21.15.186:0/2364889028 shutdown_connections 2024-03-28T15:01:35.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.768+0000 7f7dda93e640 1 -- 172.21.15.186:0/2364889028 wait complete. 2024-03-28T15:01:35.780 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T15:01:35.780 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:01:35.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.880+0000 7fb38d2d4640 1 Processor -- start 2024-03-28T15:01:35.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.880+0000 7fb38d2d4640 1 -- start start 2024-03-28T15:01:35.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.881+0000 7fb38d2d4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb388152810 0x7fb388158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.881+0000 7fb38d2d4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb388071340 0x7fb38806c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.881+0000 7fb38d2d4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb388072df0 con 0x7fb38806ce60 2024-03-28T15:01:35.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.881+0000 7fb38d2d4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb388159500 con 0x7fb388071340 2024-03-28T15:01:35.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.881+0000 7fb38d2d4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fb3880c8660 con 0x7fb388152810 2024-03-28T15:01:35.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.881+0000 7fb3867fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb388071340 0x7fb38806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.881+0000 7fb3867fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb388071340 0x7fb38806c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41184/0 (socket says 172.21.15.186:41184) 2024-03-28T15:01:35.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.881+0000 7fb3867fc640 1 -- 172.21.15.186:0/3638018939 learned_addr learned my addr 172.21.15.186:0/3638018939 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:35.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.882+0000 7fb386ffd640 1 --2- 172.21.15.186:0/3638018939 >> v2:172.21.15.191:3300/0 conn(0x7fb388152810 0x7fb388158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.882+0000 7fb3867fc640 1 -- 172.21.15.186:0/3638018939 >> v2:172.21.15.191:3300/0 conn(0x7fb388152810 msgr2=0x7fb388158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:35.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.882+0000 7fb3867fc640 1 --2- 172.21.15.186:0/3638018939 >> v2:172.21.15.191:3300/0 conn(0x7fb388152810 0x7fb388158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.882+0000 7fb3867fc640 1 -- 172.21.15.186:0/3638018939 >> v1:172.21.15.186:6789/0 conn(0x7fb38806ce60 legacy=0x7fb3880c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.882+0000 7fb3867fc640 1 -- 172.21.15.186:0/3638018939 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb388071d50 con 0x7fb388071340 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.882+0000 7fb3867fc640 1 --2- 172.21.15.186:0/3638018939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388071340 0x7fb38806c920 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009870 tx=0x7fb37c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931c86da9f432d1a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.882+0000 7fb385ffb640 1 -- 172.21.15.186:0/3638018939 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c034070 con 0x7fb388071340 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.883+0000 7fb385ffb640 1 -- 172.21.15.186:0/3638018939 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb37c030860 con 0x7fb388071340 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.883+0000 7fb385ffb640 1 -- 172.21.15.186:0/3638018939 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c030c30 con 0x7fb388071340 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.883+0000 7fb38d2d4640 1 -- 172.21.15.186:0/3638018939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388071340 msgr2=0x7fb38806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.883+0000 7fb38d2d4640 1 --2- 172.21.15.186:0/3638018939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388071340 0x7fb38806c920 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009870 tx=0x7fb37c02b9a0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.883+0000 7fb38d2d4640 1 -- 172.21.15.186:0/3638018939 shutdown_connections 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.883+0000 7fb38d2d4640 1 --2- 172.21.15.186:0/3638018939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388071340 0x7fb38806c920 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.883+0000 7fb38d2d4640 1 --2- 172.21.15.186:0/3638018939 >> v2:172.21.15.191:3300/0 conn(0x7fb388152810 0x7fb388158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.883+0000 7fb38d2d4640 1 -- 172.21.15.186:0/3638018939 >> 172.21.15.186:0/3638018939 conn(0x7fb3880c8a20 msgr2=0x7fb3880c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:35.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.883+0000 7fb38d2d4640 1 -- 172.21.15.186:0/3638018939 shutdown_connections 2024-03-28T15:01:35.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.883+0000 7fb38d2d4640 1 -- 172.21.15.186:0/3638018939 wait complete. 2024-03-28T15:01:35.889 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.884+0000 7fb38d2d4640 1 Processor -- start 2024-03-28T15:01:35.889 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.884+0000 7fb38d2d4640 1 -- start start 2024-03-28T15:01:35.889 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb38d2d4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb38806ce60 0x7fb3880c1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.889 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb38d2d4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb388071340 0x7fb3880c4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.889 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb38d2d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388152810 0x7fb3881f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb38d2d4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb3880725f0 con 0x7fb388152810 2024-03-28T15:01:35.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb38d2d4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb388072310 con 0x7fb38806ce60 2024-03-28T15:01:35.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb38d2d4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb3880728d0 con 0x7fb388071340 2024-03-28T15:01:35.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb3877fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388152810 0x7fb3881f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb3877fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388152810 0x7fb3881f4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41198/0 (socket says 172.21.15.186:41198) 2024-03-28T15:01:35.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb3877fe640 1 -- 172.21.15.186:0/2669471190 learned_addr learned my addr 172.21.15.186:0/2669471190 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:35.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb386ffd640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb38806ce60 0x7fb3880c1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb3877fe640 1 -- 172.21.15.186:0/2669471190 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb388071340 msgr2=0x7fb3880c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb3867fc640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb388071340 0x7fb3880c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb3877fe640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb388071340 0x7fb3880c4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb3877fe640 1 -- 172.21.15.186:0/2669471190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb38806ce60 msgr2=0x7fb3880c1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:35.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb3877fe640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb38806ce60 0x7fb3880c1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:35.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.885+0000 7fb3877fe640 1 -- 172.21.15.186:0/2669471190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3881e4050 con 0x7fb388152810 2024-03-28T15:01:35.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.886+0000 7fb3877fe640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388152810 0x7fb3881f4190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb38000c3f0 tx=0x7fb38000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:35.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.886+0000 7fb36f7fe640 1 -- 172.21.15.186:0/2669471190 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3800052d0 con 0x7fb388152810 2024-03-28T15:01:35.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.886+0000 7fb36f7fe640 1 -- 172.21.15.186:0/2669471190 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb380005470 con 0x7fb388152810 2024-03-28T15:01:35.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.886+0000 7fb36f7fe640 1 -- 172.21.15.186:0/2669471190 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380007260 con 0x7fb388152810 2024-03-28T15:01:35.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.886+0000 7fb38d2d4640 1 -- 172.21.15.186:0/2669471190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3881e4310 con 0x7fb388152810 2024-03-28T15:01:35.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.886+0000 7fb38d2d4640 1 -- 172.21.15.186:0/2669471190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3880c2af0 con 0x7fb388152810 2024-03-28T15:01:35.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.887+0000 7fb38d2d4640 1 -- 172.21.15.186:0/2669471190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb388069500 con 0x7fb388152810 2024-03-28T15:01:35.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.887+0000 7fb36f7fe640 1 -- 172.21.15.186:0/2669471190 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb380007400 con 0x7fb388152810 2024-03-28T15:01:35.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.887+0000 7fb36f7fe640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb37406c560 0x7fb37406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:35.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.888+0000 7fb386ffd640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb37406c560 0x7fb37406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:35.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.888+0000 7fb36f7fe640 1 -- 172.21.15.186:0/2669471190 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb38008aa00 con 0x7fb388152810 2024-03-28T15:01:35.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.888+0000 7fb386ffd640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb37406c560 0x7fb37406ea20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb378009e90 tx=0x7fb37800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:35.893 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:35.892+0000 7fb36f7fe640 1 -- 172.21.15.186:0/2669471190 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb380059170 con 0x7fb388152810 2024-03-28T15:01:36.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.064+0000 7fb38d2d4640 1 -- 172.21.15.186:0/2669471190 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb3880c77f0 con 0x7fb37406c560 2024-03-28T15:01:36.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.071+0000 7fb36f7fe640 1 -- 172.21.15.186:0/2669471190 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251848 (secure 0 0 0) 0x7fb3880c77f0 con 0x7fb37406c560 2024-03-28T15:01:36.075 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:36.077 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:01:36.078 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 -- 172.21.15.186:0/2669471190 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb37406c560 msgr2=0x7fb37406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb37406c560 0x7fb37406ea20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb378009e90 tx=0x7fb37800b040 comp rx=0 tx=0).stop 2024-03-28T15:01:36.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 -- 172.21.15.186:0/2669471190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388152810 msgr2=0x7fb3881f4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388152810 0x7fb3881f4190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb38000c3f0 tx=0x7fb38000c8b0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 -- 172.21.15.186:0/2669471190 shutdown_connections 2024-03-28T15:01:36.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb37406c560 0x7fb37406ea20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388152810 0x7fb3881f4190 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb388071340 0x7fb3880c4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 --2- 172.21.15.186:0/2669471190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb38806ce60 0x7fb3880c1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 -- 172.21.15.186:0/2669471190 >> 172.21.15.186:0/2669471190 conn(0x7fb3880c8a20 msgr2=0x7fb38814dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:36.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 -- 172.21.15.186:0/2669471190 shutdown_connections 2024-03-28T15:01:36.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.074+0000 7fb38d2d4640 1 -- 172.21.15.186:0/2669471190 wait complete. 2024-03-28T15:01:36.085 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-28T15:01:34.635547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218960,"kb_used_data":2760,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755760,"statfs":{"total":773094113280,"available":772869898240,"internally_reserved":0,"allocated":2826240,"data_stored":1748208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.561583"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461304+0000","last_change":"2024-03-28T15:01:32.122226+0000","last_active":"2024-03-28T15:01:33.461304+0000","last_peered":"2024-03-28T15:01:33.461304+0000","last_clean":"2024-03-28T15:01:33.461304+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:01:33.461304+0000","last_undegraded":"2024-03-28T15:01:33.461304+0000","last_fullsized":"2024-03-28T15:01:33.461304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104984+0000","last_change":"2024-03-28T15:01:32.115850+0000","last_active":"2024-03-28T15:01:33.104984+0000","last_peered":"2024-03-28T15:01:33.104984+0000","last_clean":"2024-03-28T15:01:33.104984+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:01:33.104984+0000","last_undegraded":"2024-03-28T15:01:33.104984+0000","last_fullsized":"2024-03-28T15:01:33.104984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.119744+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:57.022614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107952+0000","last_change":"2024-03-28T15:01:32.120916+0000","last_active":"2024-03-28T15:01:33.107952+0000","last_peered":"2024-03-28T15:01:33.107952+0000","last_clean":"2024-03-28T15:01:33.107952+0000","last_became_active":"2024-03-28T15:01:32.117449+0000","last_became_peered":"2024-03-28T15:01:32.117449+0000","last_unstale":"2024-03-28T15:01:33.107952+0000","last_undegraded":"2024-03-28T15:01:33.107952+0000","last_fullsized":"2024-03-28T15:01:33.107952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461016+0000","last_change":"2024-03-28T15:01:32.112835+0000","last_active":"2024-03-28T15:01:33.461016+0000","last_peered":"2024-03-28T15:01:33.461016+0000","last_clean":"2024-03-28T15:01:33.461016+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:01:33.461016+0000","last_undegraded":"2024-03-28T15:01:33.461016+0000","last_fullsized":"2024-03-28T15:01:33.461016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107714+0000","last_change":"2024-03-28T15:01:32.116248+0000","last_active":"2024-03-28T15:01:33.107714+0000","last_peered":"2024-03-28T15:01:33.107714+0000","last_clean":"2024-03-28T15:01:33.107714+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:01:33.107714+0000","last_undegraded":"2024-03-28T15:01:33.107714+0000","last_fullsized":"2024-03-28T15:01:33.107714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460486+0000","last_change":"2024-03-28T15:01:32.115221+0000","last_active":"2024-03-28T15:01:33.460486+0000","last_peered":"2024-03-28T15:01:33.460486+0000","last_clean":"2024-03-28T15:01:33.460486+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:01:33.460486+0000","last_undegraded":"2024-03-28T15:01:33.460486+0000","last_fullsized":"2024-03-28T15:01:33.460486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460218+0000","last_change":"2024-03-28T15:01:32.118310+0000","last_active":"2024-03-28T15:01:33.460218+0000","last_peered":"2024-03-28T15:01:33.460218+0000","last_clean":"2024-03-28T15:01:33.460218+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:01:33.460218+0000","last_undegraded":"2024-03-28T15:01:33.460218+0000","last_fullsized":"2024-03-28T15:01:33.460218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461212+0000","last_change":"2024-03-28T15:01:32.114205+0000","last_active":"2024-03-28T15:01:33.461212+0000","last_peered":"2024-03-28T15:01:33.461212+0000","last_clean":"2024-03-28T15:01:33.461212+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:01:33.461212+0000","last_undegraded":"2024-03-28T15:01:33.461212+0000","last_fullsized":"2024-03-28T15:01:33.461212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460905+0000","last_change":"2024-03-28T15:01:32.125178+0000","last_active":"2024-03-28T15:01:33.460905+0000","last_peered":"2024-03-28T15:01:33.460905+0000","last_clean":"2024-03-28T15:01:33.460905+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:01:33.460905+0000","last_undegraded":"2024-03-28T15:01:33.460905+0000","last_fullsized":"2024-03-28T15:01:33.460905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460676+0000","last_change":"2024-03-28T15:01:32.125839+0000","last_active":"2024-03-28T15:01:33.460676+0000","last_peered":"2024-03-28T15:01:33.460676+0000","last_clean":"2024-03-28T15:01:33.460676+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:01:33.460676+0000","last_undegraded":"2024-03-28T15:01:33.460676+0000","last_fullsized":"2024-03-28T15:01:33.460676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107231+0000","last_change":"2024-03-28T15:01:32.121874+0000","last_active":"2024-03-28T15:01:33.107231+0000","last_peered":"2024-03-28T15:01:33.107231+0000","last_clean":"2024-03-28T15:01:33.107231+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:01:33.107231+0000","last_undegraded":"2024-03-28T15:01:33.107231+0000","last_fullsized":"2024-03-28T15:01:33.107231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:20.744143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460735+0000","last_change":"2024-03-28T15:01:32.119348+0000","last_active":"2024-03-28T15:01:33.460735+0000","last_peered":"2024-03-28T15:01:33.460735+0000","last_clean":"2024-03-28T15:01:33.460735+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:01:33.460735+0000","last_undegraded":"2024-03-28T15:01:33.460735+0000","last_fullsized":"2024-03-28T15:01:33.460735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461204+0000","last_change":"2024-03-28T15:01:32.114290+0000","last_active":"2024-03-28T15:01:33.461204+0000","last_peered":"2024-03-28T15:01:33.461204+0000","last_clean":"2024-03-28T15:01:33.461204+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:01:33.461204+0000","last_undegraded":"2024-03-28T15:01:33.461204+0000","last_fullsized":"2024-03-28T15:01:33.461204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105071+0000","last_change":"2024-03-28T15:01:32.113654+0000","last_active":"2024-03-28T15:01:33.105071+0000","last_peered":"2024-03-28T15:01:33.105071+0000","last_clean":"2024-03-28T15:01:33.105071+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:01:33.105071+0000","last_undegraded":"2024-03-28T15:01:33.105071+0000","last_fullsized":"2024-03-28T15:01:33.105071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107620+0000","last_change":"2024-03-28T15:01:32.115901+0000","last_active":"2024-03-28T15:01:33.107620+0000","last_peered":"2024-03-28T15:01:33.107620+0000","last_clean":"2024-03-28T15:01:33.107620+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:01:33.107620+0000","last_undegraded":"2024-03-28T15:01:33.107620+0000","last_fullsized":"2024-03-28T15:01:33.107620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108153+0000","last_change":"2024-03-28T15:01:32.113156+0000","last_active":"2024-03-28T15:01:33.108153+0000","last_peered":"2024-03-28T15:01:33.108153+0000","last_clean":"2024-03-28T15:01:33.108153+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:01:33.108153+0000","last_undegraded":"2024-03-28T15:01:33.108153+0000","last_fullsized":"2024-03-28T15:01:33.108153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460557+0000","last_change":"2024-03-28T15:01:32.119785+0000","last_active":"2024-03-28T15:01:33.460557+0000","last_peered":"2024-03-28T15:01:33.460557+0000","last_clean":"2024-03-28T15:01:33.460557+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:01:33.460557+0000","last_undegraded":"2024-03-28T15:01:33.460557+0000","last_fullsized":"2024-03-28T15:01:33.460557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461426+0000","last_change":"2024-03-28T15:01:32.111912+0000","last_active":"2024-03-28T15:01:33.461426+0000","last_peered":"2024-03-28T15:01:33.461426+0000","last_clean":"2024-03-28T15:01:33.461426+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:01:33.461426+0000","last_undegraded":"2024-03-28T15:01:33.461426+0000","last_fullsized":"2024-03-28T15:01:33.461426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:52.632729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461025+0000","last_change":"2024-03-28T15:01:32.124381+0000","last_active":"2024-03-28T15:01:33.461025+0000","last_peered":"2024-03-28T15:01:33.461025+0000","last_clean":"2024-03-28T15:01:33.461025+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:01:33.461025+0000","last_undegraded":"2024-03-28T15:01:33.461025+0000","last_fullsized":"2024-03-28T15:01:33.461025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.120110+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107929+0000","last_change":"2024-03-28T15:01:32.113699+0000","last_active":"2024-03-28T15:01:33.107929+0000","last_peered":"2024-03-28T15:01:33.107929+0000","last_clean":"2024-03-28T15:01:33.107929+0000","last_became_active":"2024-03-28T15:01:32.113321+0000","last_became_peered":"2024-03-28T15:01:32.113321+0000","last_unstale":"2024-03-28T15:01:33.107929+0000","last_undegraded":"2024-03-28T15:01:33.107929+0000","last_fullsized":"2024-03-28T15:01:33.107929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460506+0000","last_change":"2024-03-28T15:01:32.113345+0000","last_active":"2024-03-28T15:01:33.460506+0000","last_peered":"2024-03-28T15:01:33.460506+0000","last_clean":"2024-03-28T15:01:33.460506+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:01:33.460506+0000","last_undegraded":"2024-03-28T15:01:33.460506+0000","last_fullsized":"2024-03-28T15:01:33.460506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107235+0000","last_change":"2024-03-28T15:01:32.112328+0000","last_active":"2024-03-28T15:01:33.107235+0000","last_peered":"2024-03-28T15:01:33.107235+0000","last_clean":"2024-03-28T15:01:33.107235+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:01:33.107235+0000","last_undegraded":"2024-03-28T15:01:33.107235+0000","last_fullsized":"2024-03-28T15:01:33.107235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461017+0000","last_change":"2024-03-28T15:01:32.122399+0000","last_active":"2024-03-28T15:01:33.461017+0000","last_peered":"2024-03-28T15:01:33.461017+0000","last_clean":"2024-03-28T15:01:33.461017+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:01:33.461017+0000","last_undegraded":"2024-03-28T15:01:33.461017+0000","last_fullsized":"2024-03-28T15:01:33.461017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460971+0000","last_change":"2024-03-28T15:01:32.123879+0000","last_active":"2024-03-28T15:01:33.460971+0000","last_peered":"2024-03-28T15:01:33.460971+0000","last_clean":"2024-03-28T15:01:33.460971+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:01:33.460971+0000","last_undegraded":"2024-03-28T15:01:33.460971+0000","last_fullsized":"2024-03-28T15:01:33.460971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104923+0000","last_change":"2024-03-28T15:01:32.110380+0000","last_active":"2024-03-28T15:01:33.104923+0000","last_peered":"2024-03-28T15:01:33.104923+0000","last_clean":"2024-03-28T15:01:33.104923+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:01:33.104923+0000","last_undegraded":"2024-03-28T15:01:33.104923+0000","last_fullsized":"2024-03-28T15:01:33.104923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460754+0000","last_change":"2024-03-28T15:01:32.113422+0000","last_active":"2024-03-28T15:01:33.460754+0000","last_peered":"2024-03-28T15:01:33.460754+0000","last_clean":"2024-03-28T15:01:33.460754+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:01:33.460754+0000","last_undegraded":"2024-03-28T15:01:33.460754+0000","last_fullsized":"2024-03-28T15:01:33.460754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.117560+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108157+0000","last_change":"2024-03-28T15:01:32.117440+0000","last_active":"2024-03-28T15:01:33.108157+0000","last_peered":"2024-03-28T15:01:33.108157+0000","last_clean":"2024-03-28T15:01:33.108157+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:01:33.108157+0000","last_undegraded":"2024-03-28T15:01:33.108157+0000","last_fullsized":"2024-03-28T15:01:33.108157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460388+0000","last_change":"2024-03-28T15:01:32.111344+0000","last_active":"2024-03-28T15:01:33.460388+0000","last_peered":"2024-03-28T15:01:33.460388+0000","last_clean":"2024-03-28T15:01:33.460388+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:01:33.460388+0000","last_undegraded":"2024-03-28T15:01:33.460388+0000","last_fullsized":"2024-03-28T15:01:33.460388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460295+0000","last_change":"2024-03-28T15:01:32.126811+0000","last_active":"2024-03-28T15:01:33.460295+0000","last_peered":"2024-03-28T15:01:33.460295+0000","last_clean":"2024-03-28T15:01:33.460295+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:01:33.460295+0000","last_undegraded":"2024-03-28T15:01:33.460295+0000","last_fullsized":"2024-03-28T15:01:33.460295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106843+0000","last_change":"2024-03-28T15:01:32.112050+0000","last_active":"2024-03-28T15:01:33.106843+0000","last_peered":"2024-03-28T15:01:33.106843+0000","last_clean":"2024-03-28T15:01:33.106843+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:01:33.106843+0000","last_undegraded":"2024-03-28T15:01:33.106843+0000","last_fullsized":"2024-03-28T15:01:33.106843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:30.376545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460853+0000","last_change":"2024-03-28T15:01:31.097415+0000","last_active":"2024-03-28T15:01:33.460853+0000","last_peered":"2024-03-28T15:01:33.460853+0000","last_clean":"2024-03-28T15:01:33.460853+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:01:33.460853+0000","last_undegraded":"2024-03-28T15:01:33.460853+0000","last_fullsized":"2024-03-28T15:01:33.460853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461379+0000","last_change":"2024-03-28T15:01:32.123479+0000","last_active":"2024-03-28T15:01:33.461379+0000","last_peered":"2024-03-28T15:01:33.461379+0000","last_clean":"2024-03-28T15:01:33.461379+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:01:33.461379+0000","last_undegraded":"2024-03-28T15:01:33.461379+0000","last_fullsized":"2024-03-28T15:01:33.461379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460265+0000","last_change":"2024-03-28T15:01:31.088437+0000","last_active":"2024-03-28T15:01:33.460265+0000","last_peered":"2024-03-28T15:01:33.460265+0000","last_clean":"2024-03-28T15:01:33.460265+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:01:33.460265+0000","last_undegraded":"2024-03-28T15:01:33.460265+0000","last_fullsized":"2024-03-28T15:01:33.460265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460403+0000","last_change":"2024-03-28T15:01:31.097499+0000","last_active":"2024-03-28T15:01:33.460403+0000","last_peered":"2024-03-28T15:01:33.460403+0000","last_clean":"2024-03-28T15:01:33.460403+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:01:33.460403+0000","last_undegraded":"2024-03-28T15:01:33.460403+0000","last_fullsized":"2024-03-28T15:01:33.460403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460343+0000","last_change":"2024-03-28T15:01:32.115727+0000","last_active":"2024-03-28T15:01:33.460343+0000","last_peered":"2024-03-28T15:01:33.460343+0000","last_clean":"2024-03-28T15:01:33.460343+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:01:33.460343+0000","last_undegraded":"2024-03-28T15:01:33.460343+0000","last_fullsized":"2024-03-28T15:01:33.460343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107881+0000","last_change":"2024-03-28T15:01:31.086252+0000","last_active":"2024-03-28T15:01:33.107881+0000","last_peered":"2024-03-28T15:01:33.107881+0000","last_clean":"2024-03-28T15:01:33.107881+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:01:33.107881+0000","last_undegraded":"2024-03-28T15:01:33.107881+0000","last_fullsized":"2024-03-28T15:01:33.107881+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104868+0000","last_change":"2024-03-28T15:01:32.110465+0000","last_active":"2024-03-28T15:01:33.104868+0000","last_peered":"2024-03-28T15:01:33.104868+0000","last_clean":"2024-03-28T15:01:33.104868+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:01:33.104868+0000","last_undegraded":"2024-03-28T15:01:33.104868+0000","last_fullsized":"2024-03-28T15:01:33.104868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460446+0000","last_change":"2024-03-28T15:01:31.092338+0000","last_active":"2024-03-28T15:01:33.460446+0000","last_peered":"2024-03-28T15:01:33.460446+0000","last_clean":"2024-03-28T15:01:33.460446+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:01:33.460446+0000","last_undegraded":"2024-03-28T15:01:33.460446+0000","last_fullsized":"2024-03-28T15:01:33.460446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107369+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.107369+0000","last_peered":"2024-03-28T15:01:33.107369+0000","last_clean":"2024-03-28T15:01:33.107369+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:01:33.107369+0000","last_undegraded":"2024-03-28T15:01:33.107369+0000","last_fullsized":"2024-03-28T15:01:33.107369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107482+0000","last_change":"2024-03-28T15:01:31.088909+0000","last_active":"2024-03-28T15:01:33.107482+0000","last_peered":"2024-03-28T15:01:33.107482+0000","last_clean":"2024-03-28T15:01:33.107482+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:01:33.107482+0000","last_undegraded":"2024-03-28T15:01:33.107482+0000","last_fullsized":"2024-03-28T15:01:33.107482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461141+0000","last_change":"2024-03-28T15:01:32.117188+0000","last_active":"2024-03-28T15:01:33.461141+0000","last_peered":"2024-03-28T15:01:33.461141+0000","last_clean":"2024-03-28T15:01:33.461141+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:01:33.461141+0000","last_undegraded":"2024-03-28T15:01:33.461141+0000","last_fullsized":"2024-03-28T15:01:33.461141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104835+0000","last_change":"2024-03-28T15:01:31.095935+0000","last_active":"2024-03-28T15:01:33.104835+0000","last_peered":"2024-03-28T15:01:33.104835+0000","last_clean":"2024-03-28T15:01:33.104835+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:01:33.104835+0000","last_undegraded":"2024-03-28T15:01:33.104835+0000","last_fullsized":"2024-03-28T15:01:33.104835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107512+0000","last_change":"2024-03-28T15:01:32.120677+0000","last_active":"2024-03-28T15:01:33.107512+0000","last_peered":"2024-03-28T15:01:33.107512+0000","last_clean":"2024-03-28T15:01:33.107512+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.107512+0000","last_undegraded":"2024-03-28T15:01:33.107512+0000","last_fullsized":"2024-03-28T15:01:33.107512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461129+0000","last_change":"2024-03-28T15:01:31.089380+0000","last_active":"2024-03-28T15:01:33.461129+0000","last_peered":"2024-03-28T15:01:33.461129+0000","last_clean":"2024-03-28T15:01:33.461129+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:01:33.461129+0000","last_undegraded":"2024-03-28T15:01:33.461129+0000","last_fullsized":"2024-03-28T15:01:33.461129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.110749+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.110332+0000","last_became_peered":"2024-03-28T15:01:32.110332+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461380+0000","last_change":"2024-03-28T15:01:31.098172+0000","last_active":"2024-03-28T15:01:33.461380+0000","last_peered":"2024-03-28T15:01:33.461380+0000","last_clean":"2024-03-28T15:01:33.461380+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:01:33.461380+0000","last_undegraded":"2024-03-28T15:01:33.461380+0000","last_fullsized":"2024-03-28T15:01:33.461380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:16.015528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461045+0000","last_change":"2024-03-28T15:01:32.117690+0000","last_active":"2024-03-28T15:01:33.461045+0000","last_peered":"2024-03-28T15:01:33.461045+0000","last_clean":"2024-03-28T15:01:33.461045+0000","last_became_active":"2024-03-28T15:01:32.117412+0000","last_became_peered":"2024-03-28T15:01:32.117412+0000","last_unstale":"2024-03-28T15:01:33.461045+0000","last_undegraded":"2024-03-28T15:01:33.461045+0000","last_fullsized":"2024-03-28T15:01:33.461045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108124+0000","last_change":"2024-03-28T15:01:31.090519+0000","last_active":"2024-03-28T15:01:33.108124+0000","last_peered":"2024-03-28T15:01:33.108124+0000","last_clean":"2024-03-28T15:01:33.108124+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:01:33.108124+0000","last_undegraded":"2024-03-28T15:01:33.108124+0000","last_fullsized":"2024-03-28T15:01:33.108124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460932+0000","last_change":"2024-03-28T15:01:32.119827+0000","last_active":"2024-03-28T15:01:33.460932+0000","last_peered":"2024-03-28T15:01:33.460932+0000","last_clean":"2024-03-28T15:01:33.460932+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:01:33.460932+0000","last_undegraded":"2024-03-28T15:01:33.460932+0000","last_fullsized":"2024-03-28T15:01:33.460932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461234+0000","last_change":"2024-03-28T15:01:31.088364+0000","last_active":"2024-03-28T15:01:33.461234+0000","last_peered":"2024-03-28T15:01:33.461234+0000","last_clean":"2024-03-28T15:01:33.461234+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:01:33.461234+0000","last_undegraded":"2024-03-28T15:01:33.461234+0000","last_fullsized":"2024-03-28T15:01:33.461234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460422+0000","last_change":"2024-03-28T15:01:32.126177+0000","last_active":"2024-03-28T15:01:33.460422+0000","last_peered":"2024-03-28T15:01:33.460422+0000","last_clean":"2024-03-28T15:01:33.460422+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:01:33.460422+0000","last_undegraded":"2024-03-28T15:01:33.460422+0000","last_fullsized":"2024-03-28T15:01:33.460422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.459994+0000","last_change":"2024-03-28T15:01:31.094837+0000","last_active":"2024-03-28T15:01:33.459994+0000","last_peered":"2024-03-28T15:01:33.459994+0000","last_clean":"2024-03-28T15:01:33.459994+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:01:33.459994+0000","last_undegraded":"2024-03-28T15:01:33.459994+0000","last_fullsized":"2024-03-28T15:01:33.459994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460675+0000","last_change":"2024-03-28T15:01:32.113803+0000","last_active":"2024-03-28T15:01:33.460675+0000","last_peered":"2024-03-28T15:01:33.460675+0000","last_clean":"2024-03-28T15:01:33.460675+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:01:33.460675+0000","last_undegraded":"2024-03-28T15:01:33.460675+0000","last_fullsized":"2024-03-28T15:01:33.460675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107517+0000","last_change":"2024-03-28T15:01:31.092455+0000","last_active":"2024-03-28T15:01:33.107517+0000","last_peered":"2024-03-28T15:01:33.107517+0000","last_clean":"2024-03-28T15:01:33.107517+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:01:33.107517+0000","last_undegraded":"2024-03-28T15:01:33.107517+0000","last_fullsized":"2024-03-28T15:01:33.107517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:04.636216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460748+0000","last_change":"2024-03-28T15:01:32.117065+0000","last_active":"2024-03-28T15:01:33.460748+0000","last_peered":"2024-03-28T15:01:33.460748+0000","last_clean":"2024-03-28T15:01:33.460748+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:01:33.460748+0000","last_undegraded":"2024-03-28T15:01:33.460748+0000","last_fullsized":"2024-03-28T15:01:33.460748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460592+0000","last_change":"2024-03-28T15:01:31.092175+0000","last_active":"2024-03-28T15:01:33.460592+0000","last_peered":"2024-03-28T15:01:33.460592+0000","last_clean":"2024-03-28T15:01:33.460592+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:01:33.460592+0000","last_undegraded":"2024-03-28T15:01:33.460592+0000","last_fullsized":"2024-03-28T15:01:33.460592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461324+0000","last_change":"2024-03-28T15:01:32.114975+0000","last_active":"2024-03-28T15:01:33.461324+0000","last_peered":"2024-03-28T15:01:33.461324+0000","last_clean":"2024-03-28T15:01:33.461324+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:01:33.461324+0000","last_undegraded":"2024-03-28T15:01:33.461324+0000","last_fullsized":"2024-03-28T15:01:33.461324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104998+0000","last_change":"2024-03-28T15:01:31.097901+0000","last_active":"2024-03-28T15:01:33.104998+0000","last_peered":"2024-03-28T15:01:33.104998+0000","last_clean":"2024-03-28T15:01:33.104998+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:01:33.104998+0000","last_undegraded":"2024-03-28T15:01:33.104998+0000","last_fullsized":"2024-03-28T15:01:33.104998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108151+0000","last_change":"2024-03-28T15:01:32.120646+0000","last_active":"2024-03-28T15:01:33.108151+0000","last_peered":"2024-03-28T15:01:33.108151+0000","last_clean":"2024-03-28T15:01:33.108151+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:01:33.108151+0000","last_undegraded":"2024-03-28T15:01:33.108151+0000","last_fullsized":"2024-03-28T15:01:33.108151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460197+0000","last_change":"2024-03-28T15:01:31.086634+0000","last_active":"2024-03-28T15:01:33.460197+0000","last_peered":"2024-03-28T15:01:33.460197+0000","last_clean":"2024-03-28T15:01:33.460197+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:01:33.460197+0000","last_undegraded":"2024-03-28T15:01:33.460197+0000","last_fullsized":"2024-03-28T15:01:33.460197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461104+0000","last_change":"2024-03-28T15:01:32.112493+0000","last_active":"2024-03-28T15:01:33.461104+0000","last_peered":"2024-03-28T15:01:33.461104+0000","last_clean":"2024-03-28T15:01:33.461104+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:01:33.461104+0000","last_undegraded":"2024-03-28T15:01:33.461104+0000","last_fullsized":"2024-03-28T15:01:33.461104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461052+0000","last_change":"2024-03-28T15:01:20.724163+0000","last_active":"2024-03-28T15:01:33.461052+0000","last_peered":"2024-03-28T15:01:33.461052+0000","last_clean":"2024-03-28T15:01:33.461052+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:01:33.461052+0000","last_undegraded":"2024-03-28T15:01:33.461052+0000","last_fullsized":"2024-03-28T15:01:33.461052+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_clean_scrub_stamp":"2024-03-28T15:01:19.548090+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:15.791893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107058+0000","last_change":"2024-03-28T15:01:31.085453+0000","last_active":"2024-03-28T15:01:33.107058+0000","last_peered":"2024-03-28T15:01:33.107058+0000","last_clean":"2024-03-28T15:01:33.107058+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:01:33.107058+0000","last_undegraded":"2024-03-28T15:01:33.107058+0000","last_fullsized":"2024-03-28T15:01:33.107058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460326+0000","last_change":"2024-03-28T15:01:32.118678+0000","last_active":"2024-03-28T15:01:33.460326+0000","last_peered":"2024-03-28T15:01:33.460326+0000","last_clean":"2024-03-28T15:01:33.460326+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:01:33.460326+0000","last_undegraded":"2024-03-28T15:01:33.460326+0000","last_fullsized":"2024-03-28T15:01:33.460326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461116+0000","last_change":"2024-03-28T15:01:31.095439+0000","last_active":"2024-03-28T15:01:33.461116+0000","last_peered":"2024-03-28T15:01:33.461116+0000","last_clean":"2024-03-28T15:01:33.461116+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:01:33.461116+0000","last_undegraded":"2024-03-28T15:01:33.461116+0000","last_fullsized":"2024-03-28T15:01:33.461116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460849+0000","last_change":"2024-03-28T15:01:32.122655+0000","last_active":"2024-03-28T15:01:33.460849+0000","last_peered":"2024-03-28T15:01:33.460849+0000","last_clean":"2024-03-28T15:01:33.460849+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:01:33.460849+0000","last_undegraded":"2024-03-28T15:01:33.460849+0000","last_fullsized":"2024-03-28T15:01:33.460849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:05.942532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460385+0000","last_change":"2024-03-28T15:01:31.093265+0000","last_active":"2024-03-28T15:01:33.460385+0000","last_peered":"2024-03-28T15:01:33.460385+0000","last_clean":"2024-03-28T15:01:33.460385+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:01:33.460385+0000","last_undegraded":"2024-03-28T15:01:33.460385+0000","last_fullsized":"2024-03-28T15:01:33.460385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104892+0000","last_change":"2024-03-28T15:01:32.108597+0000","last_active":"2024-03-28T15:01:33.104892+0000","last_peered":"2024-03-28T15:01:33.104892+0000","last_clean":"2024-03-28T15:01:33.104892+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:01:33.104892+0000","last_undegraded":"2024-03-28T15:01:33.104892+0000","last_fullsized":"2024-03-28T15:01:33.104892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108209+0000","last_change":"2024-03-28T15:01:31.094750+0000","last_active":"2024-03-28T15:01:33.108209+0000","last_peered":"2024-03-28T15:01:33.108209+0000","last_clean":"2024-03-28T15:01:33.108209+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:01:33.108209+0000","last_undegraded":"2024-03-28T15:01:33.108209+0000","last_fullsized":"2024-03-28T15:01:33.108209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106970+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.106970+0000","last_peered":"2024-03-28T15:01:33.106970+0000","last_clean":"2024-03-28T15:01:33.106970+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:01:33.106970+0000","last_undegraded":"2024-03-28T15:01:33.106970+0000","last_fullsized":"2024-03-28T15:01:33.106970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:24.747778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104874+0000","last_change":"2024-03-28T15:01:31.093706+0000","last_active":"2024-03-28T15:01:33.104874+0000","last_peered":"2024-03-28T15:01:33.104874+0000","last_clean":"2024-03-28T15:01:33.104874+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:01:33.104874+0000","last_undegraded":"2024-03-28T15:01:33.104874+0000","last_fullsized":"2024-03-28T15:01:33.104874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108112+0000","last_change":"2024-03-28T15:01:32.122840+0000","last_active":"2024-03-28T15:01:33.108112+0000","last_peered":"2024-03-28T15:01:33.108112+0000","last_clean":"2024-03-28T15:01:33.108112+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:01:33.108112+0000","last_undegraded":"2024-03-28T15:01:33.108112+0000","last_fullsized":"2024-03-28T15:01:33.108112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461371+0000","last_change":"2024-03-28T15:01:31.095126+0000","last_active":"2024-03-28T15:01:33.461371+0000","last_peered":"2024-03-28T15:01:33.461371+0000","last_clean":"2024-03-28T15:01:33.461371+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:01:33.461371+0000","last_undegraded":"2024-03-28T15:01:33.461371+0000","last_fullsized":"2024-03-28T15:01:33.461371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:21.669535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104885+0000","last_change":"2024-03-28T15:01:32.111925+0000","last_active":"2024-03-28T15:01:33.104885+0000","last_peered":"2024-03-28T15:01:33.104885+0000","last_clean":"2024-03-28T15:01:33.104885+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:01:33.104885+0000","last_undegraded":"2024-03-28T15:01:33.104885+0000","last_fullsized":"2024-03-28T15:01:33.104885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104964+0000","last_change":"2024-03-28T15:01:31.093197+0000","last_active":"2024-03-28T15:01:33.104964+0000","last_peered":"2024-03-28T15:01:33.104964+0000","last_clean":"2024-03-28T15:01:33.104964+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:01:33.104964+0000","last_undegraded":"2024-03-28T15:01:33.104964+0000","last_fullsized":"2024-03-28T15:01:33.104964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461153+0000","last_change":"2024-03-28T15:01:32.115432+0000","last_active":"2024-03-28T15:01:33.461153+0000","last_peered":"2024-03-28T15:01:33.461153+0000","last_clean":"2024-03-28T15:01:33.461153+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:01:33.461153+0000","last_undegraded":"2024-03-28T15:01:33.461153+0000","last_fullsized":"2024-03-28T15:01:33.461153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460639+0000","last_change":"2024-03-28T15:01:31.088204+0000","last_active":"2024-03-28T15:01:33.460639+0000","last_peered":"2024-03-28T15:01:33.460639+0000","last_clean":"2024-03-28T15:01:33.460639+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:01:33.460639+0000","last_undegraded":"2024-03-28T15:01:33.460639+0000","last_fullsized":"2024-03-28T15:01:33.460639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461225+0000","last_change":"2024-03-28T15:01:32.114331+0000","last_active":"2024-03-28T15:01:33.461225+0000","last_peered":"2024-03-28T15:01:33.461225+0000","last_clean":"2024-03-28T15:01:33.461225+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:01:33.461225+0000","last_undegraded":"2024-03-28T15:01:33.461225+0000","last_fullsized":"2024-03-28T15:01:33.461225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460969+0000","last_change":"2024-03-28T15:01:31.095371+0000","last_active":"2024-03-28T15:01:33.460969+0000","last_peered":"2024-03-28T15:01:33.460969+0000","last_clean":"2024-03-28T15:01:33.460969+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:01:33.460969+0000","last_undegraded":"2024-03-28T15:01:33.460969+0000","last_fullsized":"2024-03-28T15:01:33.460969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107409+0000","last_change":"2024-03-28T15:01:32.110950+0000","last_active":"2024-03-28T15:01:33.107409+0000","last_peered":"2024-03-28T15:01:33.107409+0000","last_clean":"2024-03-28T15:01:33.107409+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:01:33.107409+0000","last_undegraded":"2024-03-28T15:01:33.107409+0000","last_fullsized":"2024-03-28T15:01:33.107409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105009+0000","last_change":"2024-03-28T15:01:31.094542+0000","last_active":"2024-03-28T15:01:33.105009+0000","last_peered":"2024-03-28T15:01:33.105009+0000","last_clean":"2024-03-28T15:01:33.105009+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:01:33.105009+0000","last_undegraded":"2024-03-28T15:01:33.105009+0000","last_fullsized":"2024-03-28T15:01:33.105009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460466+0000","last_change":"2024-03-28T15:01:32.112158+0000","last_active":"2024-03-28T15:01:33.460466+0000","last_peered":"2024-03-28T15:01:33.460466+0000","last_clean":"2024-03-28T15:01:33.460466+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:01:33.460466+0000","last_undegraded":"2024-03-28T15:01:33.460466+0000","last_fullsized":"2024-03-28T15:01:33.460466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104884+0000","last_change":"2024-03-28T15:01:31.094440+0000","last_active":"2024-03-28T15:01:33.104884+0000","last_peered":"2024-03-28T15:01:33.104884+0000","last_clean":"2024-03-28T15:01:33.104884+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:01:33.104884+0000","last_undegraded":"2024-03-28T15:01:33.104884+0000","last_fullsized":"2024-03-28T15:01:33.104884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460758+0000","last_change":"2024-03-28T15:01:32.115192+0000","last_active":"2024-03-28T15:01:33.460758+0000","last_peered":"2024-03-28T15:01:33.460758+0000","last_clean":"2024-03-28T15:01:33.460758+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:01:33.460758+0000","last_undegraded":"2024-03-28T15:01:33.460758+0000","last_fullsized":"2024-03-28T15:01:33.460758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461053+0000","last_change":"2024-03-28T15:01:31.096352+0000","last_active":"2024-03-28T15:01:33.461053+0000","last_peered":"2024-03-28T15:01:33.461053+0000","last_clean":"2024-03-28T15:01:33.461053+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:01:33.461053+0000","last_undegraded":"2024-03-28T15:01:33.461053+0000","last_fullsized":"2024-03-28T15:01:33.461053+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460555+0000","last_change":"2024-03-28T15:01:32.124700+0000","last_active":"2024-03-28T15:01:33.460555+0000","last_peered":"2024-03-28T15:01:33.460555+0000","last_clean":"2024-03-28T15:01:33.460555+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:01:33.460555+0000","last_undegraded":"2024-03-28T15:01:33.460555+0000","last_fullsized":"2024-03-28T15:01:33.460555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107999+0000","last_change":"2024-03-28T15:01:32.123638+0000","last_active":"2024-03-28T15:01:33.107999+0000","last_peered":"2024-03-28T15:01:33.107999+0000","last_clean":"2024-03-28T15:01:33.107999+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:01:33.107999+0000","last_undegraded":"2024-03-28T15:01:33.107999+0000","last_fullsized":"2024-03-28T15:01:33.107999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460734+0000","last_change":"2024-03-28T15:01:31.094281+0000","last_active":"2024-03-28T15:01:33.460734+0000","last_peered":"2024-03-28T15:01:33.460734+0000","last_clean":"2024-03-28T15:01:33.460734+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:01:33.460734+0000","last_undegraded":"2024-03-28T15:01:33.460734+0000","last_fullsized":"2024-03-28T15:01:33.460734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107988+0000","last_change":"2024-03-28T15:01:32.112496+0000","last_active":"2024-03-28T15:01:33.107988+0000","last_peered":"2024-03-28T15:01:33.107988+0000","last_clean":"2024-03-28T15:01:33.107988+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:01:33.107988+0000","last_undegraded":"2024-03-28T15:01:33.107988+0000","last_fullsized":"2024-03-28T15:01:33.107988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461219+0000","last_change":"2024-03-28T15:01:31.093773+0000","last_active":"2024-03-28T15:01:33.461219+0000","last_peered":"2024-03-28T15:01:33.461219+0000","last_clean":"2024-03-28T15:01:33.461219+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:01:33.461219+0000","last_undegraded":"2024-03-28T15:01:33.461219+0000","last_fullsized":"2024-03-28T15:01:33.461219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:54.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460976+0000","last_change":"2024-03-28T15:01:32.117275+0000","last_active":"2024-03-28T15:01:33.460976+0000","last_peered":"2024-03-28T15:01:33.460976+0000","last_clean":"2024-03-28T15:01:33.460976+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:01:33.460976+0000","last_undegraded":"2024-03-28T15:01:33.460976+0000","last_fullsized":"2024-03-28T15:01:33.460976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461003+0000","last_change":"2024-03-28T15:01:31.096725+0000","last_active":"2024-03-28T15:01:33.461003+0000","last_peered":"2024-03-28T15:01:33.461003+0000","last_clean":"2024-03-28T15:01:33.461003+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:01:33.461003+0000","last_undegraded":"2024-03-28T15:01:33.461003+0000","last_fullsized":"2024-03-28T15:01:33.461003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460734+0000","last_change":"2024-03-28T15:01:32.120071+0000","last_active":"2024-03-28T15:01:33.460734+0000","last_peered":"2024-03-28T15:01:33.460734+0000","last_clean":"2024-03-28T15:01:33.460734+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:01:33.460734+0000","last_undegraded":"2024-03-28T15:01:33.460734+0000","last_fullsized":"2024-03-28T15:01:33.460734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461240+0000","last_change":"2024-03-28T15:01:31.095351+0000","last_active":"2024-03-28T15:01:33.461240+0000","last_peered":"2024-03-28T15:01:33.461240+0000","last_clean":"2024-03-28T15:01:33.461240+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:01:33.461240+0000","last_undegraded":"2024-03-28T15:01:33.461240+0000","last_fullsized":"2024-03-28T15:01:33.461240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":661302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":661302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:01:36.089 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:01:36.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.225+0000 7fe4d1908640 1 Processor -- start 2024-03-28T15:01:36.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.225+0000 7fe4d1908640 1 -- start start 2024-03-28T15:01:36.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.226+0000 7fe4d1908640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe4cc06be00 0x7fe4cc0c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.226+0000 7fe4d1908640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c9b10 0x7fe4cc0c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.226+0000 7fe4d1908640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4cc15b120 con 0x7fe4cc0c95e0 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.226+0000 7fe4d1908640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fe4cc06b540 con 0x7fe4cc06be00 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.226+0000 7fe4d1908640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fe4cc06c870 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.226+0000 7fe4cb7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c9b10 0x7fe4cc0c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.226+0000 7fe4caffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe4cc06be00 0x7fe4cc0c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.226+0000 7fe4cb7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c9b10 0x7fe4cc0c9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:56782/0 (socket says 172.21.15.186:56782) 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.226+0000 7fe4cb7fe640 1 -- 172.21.15.186:0/2147906361 learned_addr learned my addr 172.21.15.186:0/2147906361 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.227+0000 7fe4cb7fe640 1 -- 172.21.15.186:0/2147906361 >> v2:172.21.15.191:3300/0 conn(0x7fe4cc06be00 msgr2=0x7fe4cc0c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.227+0000 7fe4cb7fe640 1 --2- 172.21.15.186:0/2147906361 >> v2:172.21.15.191:3300/0 conn(0x7fe4cc06be00 0x7fe4cc0c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.227+0000 7fe4cb7fe640 1 -- 172.21.15.186:0/2147906361 >> v1:172.21.15.191:6789/0 conn(0x7fe4cc0c95e0 legacy=0x7fe4cc15aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.227+0000 7fe4cb7fe640 1 -- 172.21.15.186:0/2147906361 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4cc06cb50 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.227+0000 7fe4cb7fe640 1 --2- 172.21.15.186:0/2147906361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c9b10 0x7fe4cc0c9fa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009870 tx=0x7fe4c402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=516263b45f7789c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:36.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.227+0000 7fe4c9ffb640 1 -- 172.21.15.186:0/2147906361 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c402f840 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.227+0000 7fe4c9ffb640 1 -- 172.21.15.186:0/2147906361 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c402f9e0 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.227+0000 7fe4c9ffb640 1 -- 172.21.15.186:0/2147906361 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c402fdb0 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.228+0000 7fe4d1908640 1 -- 172.21.15.186:0/2147906361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c9b10 msgr2=0x7fe4cc0c9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.228+0000 7fe4d1908640 1 --2- 172.21.15.186:0/2147906361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c9b10 0x7fe4cc0c9fa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009870 tx=0x7fe4c402ba20 comp rx=0 tx=0).stop 2024-03-28T15:01:36.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.228+0000 7fe4d1908640 1 -- 172.21.15.186:0/2147906361 shutdown_connections 2024-03-28T15:01:36.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.228+0000 7fe4d1908640 1 --2- 172.21.15.186:0/2147906361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c9b10 0x7fe4cc0c9fa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.228+0000 7fe4d1908640 1 --2- 172.21.15.186:0/2147906361 >> v2:172.21.15.191:3300/0 conn(0x7fe4cc06be00 0x7fe4cc0c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.228+0000 7fe4d1908640 1 -- 172.21.15.186:0/2147906361 >> 172.21.15.186:0/2147906361 conn(0x7fe4cc06e970 msgr2=0x7fe4cc06ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:36.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.228+0000 7fe4d1908640 1 -- 172.21.15.186:0/2147906361 shutdown_connections 2024-03-28T15:01:36.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.228+0000 7fe4d1908640 1 -- 172.21.15.186:0/2147906361 wait complete. 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.229+0000 7fe4d1908640 1 Processor -- start 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.229+0000 7fe4d1908640 1 -- start start 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4d1908640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4cc06be00 0x7fe4cc0af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4d1908640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c95e0 0x7fe4cc0c2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4d1908640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4cc0c9b10 0x7fe4cc151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4d1908640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe4cc15e540 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4d1908640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe4cc15e260 con 0x7fe4cc06be00 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4d1908640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fe4cc15e820 con 0x7fe4cc0c95e0 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4cb7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4cc0c9b10 0x7fe4cc151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4cb7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4cc0c9b10 0x7fe4cc151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41208/0 (socket says 172.21.15.186:41208) 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4cb7fe640 1 -- 172.21.15.186:0/1227702953 learned_addr learned my addr 172.21.15.186:0/1227702953 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4ca7fc640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c95e0 0x7fe4cc0c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4caffd640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4cc06be00 0x7fe4cc0af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4cb7fe640 1 -- 172.21.15.186:0/1227702953 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c95e0 msgr2=0x7fe4cc0c2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4cb7fe640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c95e0 0x7fe4cc0c2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4cb7fe640 1 -- 172.21.15.186:0/1227702953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4cc06be00 msgr2=0x7fe4cc0af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4cb7fe640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4cc06be00 0x7fe4cc0af8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4cb7fe640 1 -- 172.21.15.186:0/1227702953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4cc1f1360 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.230+0000 7fe4caffd640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4cc06be00 0x7fe4cc0af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.231+0000 7fe4cb7fe640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4cc0c9b10 0x7fe4cc151770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe4c402ba60 tx=0x7fe4c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.231+0000 7fe4b3fff640 1 -- 172.21.15.186:0/1227702953 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4047070 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.231+0000 7fe4b3fff640 1 -- 172.21.15.186:0/1227702953 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c4037c60 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.231+0000 7fe4d1908640 1 -- 172.21.15.186:0/1227702953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4cc159b30 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.231+0000 7fe4b3fff640 1 -- 172.21.15.186:0/1227702953 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40426c0 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.231+0000 7fe4d1908640 1 -- 172.21.15.186:0/1227702953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4cc15a0f0 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.232+0000 7fe4d1908640 1 -- 172.21.15.186:0/1227702953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4cc073910 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.232+0000 7fe4b3fff640 1 -- 172.21.15.186:0/1227702953 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe4c404a070 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.233+0000 7fe4b3fff640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe4b806c530 0x7fe4b806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.233+0000 7fe4caffd640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe4b806c530 0x7fe4b806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.233+0000 7fe4b3fff640 1 -- 172.21.15.186:0/1227702953 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe4c40ad840 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.233+0000 7fe4caffd640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe4b806c530 0x7fe4b806e9f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0074c0 tx=0x7fe4bc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:36.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.237+0000 7fe4b3fff640 1 -- 172.21.15.186:0/1227702953 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fe4cc073910 con 0x7fe4cc0c9b10 2024-03-28T15:01:36.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.409+0000 7fe4d1908640 1 -- 172.21.15.186:0/1227702953 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe4cc06c310 con 0x7fe4b806c530 2024-03-28T15:01:36.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.414+0000 7fe4b3fff640 1 -- 172.21.15.186:0/1227702953 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251848 (secure 0 0 0) 0x7fe4cc06c310 con 0x7fe4b806c530 2024-03-28T15:01:36.417 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:36.418 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.417+0000 7fe4d1908640 1 -- 172.21.15.186:0/1227702953 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe4b806c530 msgr2=0x7fe4b806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.417+0000 7fe4d1908640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe4b806c530 0x7fe4b806e9f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0074c0 tx=0x7fe4bc00b040 comp rx=0 tx=0).stop 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.417+0000 7fe4d1908640 1 -- 172.21.15.186:0/1227702953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4cc0c9b10 msgr2=0x7fe4cc151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.417+0000 7fe4d1908640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4cc0c9b10 0x7fe4cc151770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe4c402ba60 tx=0x7fe4c4004730 comp rx=0 tx=0).stop 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.417+0000 7fe4d1908640 1 -- 172.21.15.186:0/1227702953 shutdown_connections 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.417+0000 7fe4d1908640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe4b806c530 0x7fe4b806e9f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.417+0000 7fe4d1908640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4cc0c9b10 0x7fe4cc151770 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.417+0000 7fe4d1908640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4cc0c95e0 0x7fe4cc0c2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.417+0000 7fe4d1908640 1 --2- 172.21.15.186:0/1227702953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4cc06be00 0x7fe4cc0af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.417+0000 7fe4d1908640 1 -- 172.21.15.186:0/1227702953 >> 172.21.15.186:0/1227702953 conn(0x7fe4cc06e970 msgr2=0x7fe4cc0723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.418+0000 7fe4d1908640 1 -- 172.21.15.186:0/1227702953 shutdown_connections 2024-03-28T15:01:36.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.418+0000 7fe4d1908640 1 -- 172.21.15.186:0/1227702953 wait complete. 2024-03-28T15:01:36.428 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-28T15:01:34.635547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218960,"kb_used_data":2760,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755760,"statfs":{"total":773094113280,"available":772869898240,"internally_reserved":0,"allocated":2826240,"data_stored":1748208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.561583"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461304+0000","last_change":"2024-03-28T15:01:32.122226+0000","last_active":"2024-03-28T15:01:33.461304+0000","last_peered":"2024-03-28T15:01:33.461304+0000","last_clean":"2024-03-28T15:01:33.461304+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:01:33.461304+0000","last_undegraded":"2024-03-28T15:01:33.461304+0000","last_fullsized":"2024-03-28T15:01:33.461304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104984+0000","last_change":"2024-03-28T15:01:32.115850+0000","last_active":"2024-03-28T15:01:33.104984+0000","last_peered":"2024-03-28T15:01:33.104984+0000","last_clean":"2024-03-28T15:01:33.104984+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:01:33.104984+0000","last_undegraded":"2024-03-28T15:01:33.104984+0000","last_fullsized":"2024-03-28T15:01:33.104984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.119744+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:57.022614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107952+0000","last_change":"2024-03-28T15:01:32.120916+0000","last_active":"2024-03-28T15:01:33.107952+0000","last_peered":"2024-03-28T15:01:33.107952+0000","last_clean":"2024-03-28T15:01:33.107952+0000","last_became_active":"2024-03-28T15:01:32.117449+0000","last_became_peered":"2024-03-28T15:01:32.117449+0000","last_unstale":"2024-03-28T15:01:33.107952+0000","last_undegraded":"2024-03-28T15:01:33.107952+0000","last_fullsized":"2024-03-28T15:01:33.107952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461016+0000","last_change":"2024-03-28T15:01:32.112835+0000","last_active":"2024-03-28T15:01:33.461016+0000","last_peered":"2024-03-28T15:01:33.461016+0000","last_clean":"2024-03-28T15:01:33.461016+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:01:33.461016+0000","last_undegraded":"2024-03-28T15:01:33.461016+0000","last_fullsized":"2024-03-28T15:01:33.461016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107714+0000","last_change":"2024-03-28T15:01:32.116248+0000","last_active":"2024-03-28T15:01:33.107714+0000","last_peered":"2024-03-28T15:01:33.107714+0000","last_clean":"2024-03-28T15:01:33.107714+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:01:33.107714+0000","last_undegraded":"2024-03-28T15:01:33.107714+0000","last_fullsized":"2024-03-28T15:01:33.107714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460486+0000","last_change":"2024-03-28T15:01:32.115221+0000","last_active":"2024-03-28T15:01:33.460486+0000","last_peered":"2024-03-28T15:01:33.460486+0000","last_clean":"2024-03-28T15:01:33.460486+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:01:33.460486+0000","last_undegraded":"2024-03-28T15:01:33.460486+0000","last_fullsized":"2024-03-28T15:01:33.460486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460218+0000","last_change":"2024-03-28T15:01:32.118310+0000","last_active":"2024-03-28T15:01:33.460218+0000","last_peered":"2024-03-28T15:01:33.460218+0000","last_clean":"2024-03-28T15:01:33.460218+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:01:33.460218+0000","last_undegraded":"2024-03-28T15:01:33.460218+0000","last_fullsized":"2024-03-28T15:01:33.460218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461212+0000","last_change":"2024-03-28T15:01:32.114205+0000","last_active":"2024-03-28T15:01:33.461212+0000","last_peered":"2024-03-28T15:01:33.461212+0000","last_clean":"2024-03-28T15:01:33.461212+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:01:33.461212+0000","last_undegraded":"2024-03-28T15:01:33.461212+0000","last_fullsized":"2024-03-28T15:01:33.461212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460905+0000","last_change":"2024-03-28T15:01:32.125178+0000","last_active":"2024-03-28T15:01:33.460905+0000","last_peered":"2024-03-28T15:01:33.460905+0000","last_clean":"2024-03-28T15:01:33.460905+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:01:33.460905+0000","last_undegraded":"2024-03-28T15:01:33.460905+0000","last_fullsized":"2024-03-28T15:01:33.460905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460676+0000","last_change":"2024-03-28T15:01:32.125839+0000","last_active":"2024-03-28T15:01:33.460676+0000","last_peered":"2024-03-28T15:01:33.460676+0000","last_clean":"2024-03-28T15:01:33.460676+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:01:33.460676+0000","last_undegraded":"2024-03-28T15:01:33.460676+0000","last_fullsized":"2024-03-28T15:01:33.460676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107231+0000","last_change":"2024-03-28T15:01:32.121874+0000","last_active":"2024-03-28T15:01:33.107231+0000","last_peered":"2024-03-28T15:01:33.107231+0000","last_clean":"2024-03-28T15:01:33.107231+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:01:33.107231+0000","last_undegraded":"2024-03-28T15:01:33.107231+0000","last_fullsized":"2024-03-28T15:01:33.107231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:20.744143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460735+0000","last_change":"2024-03-28T15:01:32.119348+0000","last_active":"2024-03-28T15:01:33.460735+0000","last_peered":"2024-03-28T15:01:33.460735+0000","last_clean":"2024-03-28T15:01:33.460735+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:01:33.460735+0000","last_undegraded":"2024-03-28T15:01:33.460735+0000","last_fullsized":"2024-03-28T15:01:33.460735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461204+0000","last_change":"2024-03-28T15:01:32.114290+0000","last_active":"2024-03-28T15:01:33.461204+0000","last_peered":"2024-03-28T15:01:33.461204+0000","last_clean":"2024-03-28T15:01:33.461204+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:01:33.461204+0000","last_undegraded":"2024-03-28T15:01:33.461204+0000","last_fullsized":"2024-03-28T15:01:33.461204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105071+0000","last_change":"2024-03-28T15:01:32.113654+0000","last_active":"2024-03-28T15:01:33.105071+0000","last_peered":"2024-03-28T15:01:33.105071+0000","last_clean":"2024-03-28T15:01:33.105071+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:01:33.105071+0000","last_undegraded":"2024-03-28T15:01:33.105071+0000","last_fullsized":"2024-03-28T15:01:33.105071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107620+0000","last_change":"2024-03-28T15:01:32.115901+0000","last_active":"2024-03-28T15:01:33.107620+0000","last_peered":"2024-03-28T15:01:33.107620+0000","last_clean":"2024-03-28T15:01:33.107620+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:01:33.107620+0000","last_undegraded":"2024-03-28T15:01:33.107620+0000","last_fullsized":"2024-03-28T15:01:33.107620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108153+0000","last_change":"2024-03-28T15:01:32.113156+0000","last_active":"2024-03-28T15:01:33.108153+0000","last_peered":"2024-03-28T15:01:33.108153+0000","last_clean":"2024-03-28T15:01:33.108153+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:01:33.108153+0000","last_undegraded":"2024-03-28T15:01:33.108153+0000","last_fullsized":"2024-03-28T15:01:33.108153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460557+0000","last_change":"2024-03-28T15:01:32.119785+0000","last_active":"2024-03-28T15:01:33.460557+0000","last_peered":"2024-03-28T15:01:33.460557+0000","last_clean":"2024-03-28T15:01:33.460557+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:01:33.460557+0000","last_undegraded":"2024-03-28T15:01:33.460557+0000","last_fullsized":"2024-03-28T15:01:33.460557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461426+0000","last_change":"2024-03-28T15:01:32.111912+0000","last_active":"2024-03-28T15:01:33.461426+0000","last_peered":"2024-03-28T15:01:33.461426+0000","last_clean":"2024-03-28T15:01:33.461426+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:01:33.461426+0000","last_undegraded":"2024-03-28T15:01:33.461426+0000","last_fullsized":"2024-03-28T15:01:33.461426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:52.632729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461025+0000","last_change":"2024-03-28T15:01:32.124381+0000","last_active":"2024-03-28T15:01:33.461025+0000","last_peered":"2024-03-28T15:01:33.461025+0000","last_clean":"2024-03-28T15:01:33.461025+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:01:33.461025+0000","last_undegraded":"2024-03-28T15:01:33.461025+0000","last_fullsized":"2024-03-28T15:01:33.461025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.120110+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107929+0000","last_change":"2024-03-28T15:01:32.113699+0000","last_active":"2024-03-28T15:01:33.107929+0000","last_peered":"2024-03-28T15:01:33.107929+0000","last_clean":"2024-03-28T15:01:33.107929+0000","last_became_active":"2024-03-28T15:01:32.113321+0000","last_became_peered":"2024-03-28T15:01:32.113321+0000","last_unstale":"2024-03-28T15:01:33.107929+0000","last_undegraded":"2024-03-28T15:01:33.107929+0000","last_fullsized":"2024-03-28T15:01:33.107929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460506+0000","last_change":"2024-03-28T15:01:32.113345+0000","last_active":"2024-03-28T15:01:33.460506+0000","last_peered":"2024-03-28T15:01:33.460506+0000","last_clean":"2024-03-28T15:01:33.460506+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:01:33.460506+0000","last_undegraded":"2024-03-28T15:01:33.460506+0000","last_fullsized":"2024-03-28T15:01:33.460506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107235+0000","last_change":"2024-03-28T15:01:32.112328+0000","last_active":"2024-03-28T15:01:33.107235+0000","last_peered":"2024-03-28T15:01:33.107235+0000","last_clean":"2024-03-28T15:01:33.107235+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:01:33.107235+0000","last_undegraded":"2024-03-28T15:01:33.107235+0000","last_fullsized":"2024-03-28T15:01:33.107235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461017+0000","last_change":"2024-03-28T15:01:32.122399+0000","last_active":"2024-03-28T15:01:33.461017+0000","last_peered":"2024-03-28T15:01:33.461017+0000","last_clean":"2024-03-28T15:01:33.461017+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:01:33.461017+0000","last_undegraded":"2024-03-28T15:01:33.461017+0000","last_fullsized":"2024-03-28T15:01:33.461017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460971+0000","last_change":"2024-03-28T15:01:32.123879+0000","last_active":"2024-03-28T15:01:33.460971+0000","last_peered":"2024-03-28T15:01:33.460971+0000","last_clean":"2024-03-28T15:01:33.460971+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:01:33.460971+0000","last_undegraded":"2024-03-28T15:01:33.460971+0000","last_fullsized":"2024-03-28T15:01:33.460971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104923+0000","last_change":"2024-03-28T15:01:32.110380+0000","last_active":"2024-03-28T15:01:33.104923+0000","last_peered":"2024-03-28T15:01:33.104923+0000","last_clean":"2024-03-28T15:01:33.104923+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:01:33.104923+0000","last_undegraded":"2024-03-28T15:01:33.104923+0000","last_fullsized":"2024-03-28T15:01:33.104923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460754+0000","last_change":"2024-03-28T15:01:32.113422+0000","last_active":"2024-03-28T15:01:33.460754+0000","last_peered":"2024-03-28T15:01:33.460754+0000","last_clean":"2024-03-28T15:01:33.460754+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:01:33.460754+0000","last_undegraded":"2024-03-28T15:01:33.460754+0000","last_fullsized":"2024-03-28T15:01:33.460754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.117560+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108157+0000","last_change":"2024-03-28T15:01:32.117440+0000","last_active":"2024-03-28T15:01:33.108157+0000","last_peered":"2024-03-28T15:01:33.108157+0000","last_clean":"2024-03-28T15:01:33.108157+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:01:33.108157+0000","last_undegraded":"2024-03-28T15:01:33.108157+0000","last_fullsized":"2024-03-28T15:01:33.108157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460388+0000","last_change":"2024-03-28T15:01:32.111344+0000","last_active":"2024-03-28T15:01:33.460388+0000","last_peered":"2024-03-28T15:01:33.460388+0000","last_clean":"2024-03-28T15:01:33.460388+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:01:33.460388+0000","last_undegraded":"2024-03-28T15:01:33.460388+0000","last_fullsized":"2024-03-28T15:01:33.460388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460295+0000","last_change":"2024-03-28T15:01:32.126811+0000","last_active":"2024-03-28T15:01:33.460295+0000","last_peered":"2024-03-28T15:01:33.460295+0000","last_clean":"2024-03-28T15:01:33.460295+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:01:33.460295+0000","last_undegraded":"2024-03-28T15:01:33.460295+0000","last_fullsized":"2024-03-28T15:01:33.460295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106843+0000","last_change":"2024-03-28T15:01:32.112050+0000","last_active":"2024-03-28T15:01:33.106843+0000","last_peered":"2024-03-28T15:01:33.106843+0000","last_clean":"2024-03-28T15:01:33.106843+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:01:33.106843+0000","last_undegraded":"2024-03-28T15:01:33.106843+0000","last_fullsized":"2024-03-28T15:01:33.106843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:30.376545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460853+0000","last_change":"2024-03-28T15:01:31.097415+0000","last_active":"2024-03-28T15:01:33.460853+0000","last_peered":"2024-03-28T15:01:33.460853+0000","last_clean":"2024-03-28T15:01:33.460853+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:01:33.460853+0000","last_undegraded":"2024-03-28T15:01:33.460853+0000","last_fullsized":"2024-03-28T15:01:33.460853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461379+0000","last_change":"2024-03-28T15:01:32.123479+0000","last_active":"2024-03-28T15:01:33.461379+0000","last_peered":"2024-03-28T15:01:33.461379+0000","last_clean":"2024-03-28T15:01:33.461379+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:01:33.461379+0000","last_undegraded":"2024-03-28T15:01:33.461379+0000","last_fullsized":"2024-03-28T15:01:33.461379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460265+0000","last_change":"2024-03-28T15:01:31.088437+0000","last_active":"2024-03-28T15:01:33.460265+0000","last_peered":"2024-03-28T15:01:33.460265+0000","last_clean":"2024-03-28T15:01:33.460265+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:01:33.460265+0000","last_undegraded":"2024-03-28T15:01:33.460265+0000","last_fullsized":"2024-03-28T15:01:33.460265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460403+0000","last_change":"2024-03-28T15:01:31.097499+0000","last_active":"2024-03-28T15:01:33.460403+0000","last_peered":"2024-03-28T15:01:33.460403+0000","last_clean":"2024-03-28T15:01:33.460403+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:01:33.460403+0000","last_undegraded":"2024-03-28T15:01:33.460403+0000","last_fullsized":"2024-03-28T15:01:33.460403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460343+0000","last_change":"2024-03-28T15:01:32.115727+0000","last_active":"2024-03-28T15:01:33.460343+0000","last_peered":"2024-03-28T15:01:33.460343+0000","last_clean":"2024-03-28T15:01:33.460343+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:01:33.460343+0000","last_undegraded":"2024-03-28T15:01:33.460343+0000","last_fullsized":"2024-03-28T15:01:33.460343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107881+0000","last_change":"2024-03-28T15:01:31.086252+0000","last_active":"2024-03-28T15:01:33.107881+0000","last_peered":"2024-03-28T15:01:33.107881+0000","last_clean":"2024-03-28T15:01:33.107881+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:01:33.107881+0000","last_undegraded":"2024-03-28T15:01:33.107881+0000","last_fullsized":"2024-03-28T15:01:33.107881+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104868+0000","last_change":"2024-03-28T15:01:32.110465+0000","last_active":"2024-03-28T15:01:33.104868+0000","last_peered":"2024-03-28T15:01:33.104868+0000","last_clean":"2024-03-28T15:01:33.104868+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:01:33.104868+0000","last_undegraded":"2024-03-28T15:01:33.104868+0000","last_fullsized":"2024-03-28T15:01:33.104868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460446+0000","last_change":"2024-03-28T15:01:31.092338+0000","last_active":"2024-03-28T15:01:33.460446+0000","last_peered":"2024-03-28T15:01:33.460446+0000","last_clean":"2024-03-28T15:01:33.460446+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:01:33.460446+0000","last_undegraded":"2024-03-28T15:01:33.460446+0000","last_fullsized":"2024-03-28T15:01:33.460446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107369+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.107369+0000","last_peered":"2024-03-28T15:01:33.107369+0000","last_clean":"2024-03-28T15:01:33.107369+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:01:33.107369+0000","last_undegraded":"2024-03-28T15:01:33.107369+0000","last_fullsized":"2024-03-28T15:01:33.107369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107482+0000","last_change":"2024-03-28T15:01:31.088909+0000","last_active":"2024-03-28T15:01:33.107482+0000","last_peered":"2024-03-28T15:01:33.107482+0000","last_clean":"2024-03-28T15:01:33.107482+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:01:33.107482+0000","last_undegraded":"2024-03-28T15:01:33.107482+0000","last_fullsized":"2024-03-28T15:01:33.107482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461141+0000","last_change":"2024-03-28T15:01:32.117188+0000","last_active":"2024-03-28T15:01:33.461141+0000","last_peered":"2024-03-28T15:01:33.461141+0000","last_clean":"2024-03-28T15:01:33.461141+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:01:33.461141+0000","last_undegraded":"2024-03-28T15:01:33.461141+0000","last_fullsized":"2024-03-28T15:01:33.461141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104835+0000","last_change":"2024-03-28T15:01:31.095935+0000","last_active":"2024-03-28T15:01:33.104835+0000","last_peered":"2024-03-28T15:01:33.104835+0000","last_clean":"2024-03-28T15:01:33.104835+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:01:33.104835+0000","last_undegraded":"2024-03-28T15:01:33.104835+0000","last_fullsized":"2024-03-28T15:01:33.104835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107512+0000","last_change":"2024-03-28T15:01:32.120677+0000","last_active":"2024-03-28T15:01:33.107512+0000","last_peered":"2024-03-28T15:01:33.107512+0000","last_clean":"2024-03-28T15:01:33.107512+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.107512+0000","last_undegraded":"2024-03-28T15:01:33.107512+0000","last_fullsized":"2024-03-28T15:01:33.107512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461129+0000","last_change":"2024-03-28T15:01:31.089380+0000","last_active":"2024-03-28T15:01:33.461129+0000","last_peered":"2024-03-28T15:01:33.461129+0000","last_clean":"2024-03-28T15:01:33.461129+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:01:33.461129+0000","last_undegraded":"2024-03-28T15:01:33.461129+0000","last_fullsized":"2024-03-28T15:01:33.461129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.110749+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.110332+0000","last_became_peered":"2024-03-28T15:01:32.110332+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461380+0000","last_change":"2024-03-28T15:01:31.098172+0000","last_active":"2024-03-28T15:01:33.461380+0000","last_peered":"2024-03-28T15:01:33.461380+0000","last_clean":"2024-03-28T15:01:33.461380+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:01:33.461380+0000","last_undegraded":"2024-03-28T15:01:33.461380+0000","last_fullsized":"2024-03-28T15:01:33.461380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:16.015528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461045+0000","last_change":"2024-03-28T15:01:32.117690+0000","last_active":"2024-03-28T15:01:33.461045+0000","last_peered":"2024-03-28T15:01:33.461045+0000","last_clean":"2024-03-28T15:01:33.461045+0000","last_became_active":"2024-03-28T15:01:32.117412+0000","last_became_peered":"2024-03-28T15:01:32.117412+0000","last_unstale":"2024-03-28T15:01:33.461045+0000","last_undegraded":"2024-03-28T15:01:33.461045+0000","last_fullsized":"2024-03-28T15:01:33.461045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108124+0000","last_change":"2024-03-28T15:01:31.090519+0000","last_active":"2024-03-28T15:01:33.108124+0000","last_peered":"2024-03-28T15:01:33.108124+0000","last_clean":"2024-03-28T15:01:33.108124+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:01:33.108124+0000","last_undegraded":"2024-03-28T15:01:33.108124+0000","last_fullsized":"2024-03-28T15:01:33.108124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460932+0000","last_change":"2024-03-28T15:01:32.119827+0000","last_active":"2024-03-28T15:01:33.460932+0000","last_peered":"2024-03-28T15:01:33.460932+0000","last_clean":"2024-03-28T15:01:33.460932+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:01:33.460932+0000","last_undegraded":"2024-03-28T15:01:33.460932+0000","last_fullsized":"2024-03-28T15:01:33.460932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461234+0000","last_change":"2024-03-28T15:01:31.088364+0000","last_active":"2024-03-28T15:01:33.461234+0000","last_peered":"2024-03-28T15:01:33.461234+0000","last_clean":"2024-03-28T15:01:33.461234+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:01:33.461234+0000","last_undegraded":"2024-03-28T15:01:33.461234+0000","last_fullsized":"2024-03-28T15:01:33.461234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460422+0000","last_change":"2024-03-28T15:01:32.126177+0000","last_active":"2024-03-28T15:01:33.460422+0000","last_peered":"2024-03-28T15:01:33.460422+0000","last_clean":"2024-03-28T15:01:33.460422+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:01:33.460422+0000","last_undegraded":"2024-03-28T15:01:33.460422+0000","last_fullsized":"2024-03-28T15:01:33.460422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.459994+0000","last_change":"2024-03-28T15:01:31.094837+0000","last_active":"2024-03-28T15:01:33.459994+0000","last_peered":"2024-03-28T15:01:33.459994+0000","last_clean":"2024-03-28T15:01:33.459994+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:01:33.459994+0000","last_undegraded":"2024-03-28T15:01:33.459994+0000","last_fullsized":"2024-03-28T15:01:33.459994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460675+0000","last_change":"2024-03-28T15:01:32.113803+0000","last_active":"2024-03-28T15:01:33.460675+0000","last_peered":"2024-03-28T15:01:33.460675+0000","last_clean":"2024-03-28T15:01:33.460675+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:01:33.460675+0000","last_undegraded":"2024-03-28T15:01:33.460675+0000","last_fullsized":"2024-03-28T15:01:33.460675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107517+0000","last_change":"2024-03-28T15:01:31.092455+0000","last_active":"2024-03-28T15:01:33.107517+0000","last_peered":"2024-03-28T15:01:33.107517+0000","last_clean":"2024-03-28T15:01:33.107517+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:01:33.107517+0000","last_undegraded":"2024-03-28T15:01:33.107517+0000","last_fullsized":"2024-03-28T15:01:33.107517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:04.636216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460748+0000","last_change":"2024-03-28T15:01:32.117065+0000","last_active":"2024-03-28T15:01:33.460748+0000","last_peered":"2024-03-28T15:01:33.460748+0000","last_clean":"2024-03-28T15:01:33.460748+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:01:33.460748+0000","last_undegraded":"2024-03-28T15:01:33.460748+0000","last_fullsized":"2024-03-28T15:01:33.460748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460592+0000","last_change":"2024-03-28T15:01:31.092175+0000","last_active":"2024-03-28T15:01:33.460592+0000","last_peered":"2024-03-28T15:01:33.460592+0000","last_clean":"2024-03-28T15:01:33.460592+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:01:33.460592+0000","last_undegraded":"2024-03-28T15:01:33.460592+0000","last_fullsized":"2024-03-28T15:01:33.460592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461324+0000","last_change":"2024-03-28T15:01:32.114975+0000","last_active":"2024-03-28T15:01:33.461324+0000","last_peered":"2024-03-28T15:01:33.461324+0000","last_clean":"2024-03-28T15:01:33.461324+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:01:33.461324+0000","last_undegraded":"2024-03-28T15:01:33.461324+0000","last_fullsized":"2024-03-28T15:01:33.461324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104998+0000","last_change":"2024-03-28T15:01:31.097901+0000","last_active":"2024-03-28T15:01:33.104998+0000","last_peered":"2024-03-28T15:01:33.104998+0000","last_clean":"2024-03-28T15:01:33.104998+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:01:33.104998+0000","last_undegraded":"2024-03-28T15:01:33.104998+0000","last_fullsized":"2024-03-28T15:01:33.104998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108151+0000","last_change":"2024-03-28T15:01:32.120646+0000","last_active":"2024-03-28T15:01:33.108151+0000","last_peered":"2024-03-28T15:01:33.108151+0000","last_clean":"2024-03-28T15:01:33.108151+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:01:33.108151+0000","last_undegraded":"2024-03-28T15:01:33.108151+0000","last_fullsized":"2024-03-28T15:01:33.108151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460197+0000","last_change":"2024-03-28T15:01:31.086634+0000","last_active":"2024-03-28T15:01:33.460197+0000","last_peered":"2024-03-28T15:01:33.460197+0000","last_clean":"2024-03-28T15:01:33.460197+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:01:33.460197+0000","last_undegraded":"2024-03-28T15:01:33.460197+0000","last_fullsized":"2024-03-28T15:01:33.460197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461104+0000","last_change":"2024-03-28T15:01:32.112493+0000","last_active":"2024-03-28T15:01:33.461104+0000","last_peered":"2024-03-28T15:01:33.461104+0000","last_clean":"2024-03-28T15:01:33.461104+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:01:33.461104+0000","last_undegraded":"2024-03-28T15:01:33.461104+0000","last_fullsized":"2024-03-28T15:01:33.461104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461052+0000","last_change":"2024-03-28T15:01:20.724163+0000","last_active":"2024-03-28T15:01:33.461052+0000","last_peered":"2024-03-28T15:01:33.461052+0000","last_clean":"2024-03-28T15:01:33.461052+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:01:33.461052+0000","last_undegraded":"2024-03-28T15:01:33.461052+0000","last_fullsized":"2024-03-28T15:01:33.461052+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_clean_scrub_stamp":"2024-03-28T15:01:19.548090+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:15.791893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107058+0000","last_change":"2024-03-28T15:01:31.085453+0000","last_active":"2024-03-28T15:01:33.107058+0000","last_peered":"2024-03-28T15:01:33.107058+0000","last_clean":"2024-03-28T15:01:33.107058+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:01:33.107058+0000","last_undegraded":"2024-03-28T15:01:33.107058+0000","last_fullsized":"2024-03-28T15:01:33.107058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460326+0000","last_change":"2024-03-28T15:01:32.118678+0000","last_active":"2024-03-28T15:01:33.460326+0000","last_peered":"2024-03-28T15:01:33.460326+0000","last_clean":"2024-03-28T15:01:33.460326+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:01:33.460326+0000","last_undegraded":"2024-03-28T15:01:33.460326+0000","last_fullsized":"2024-03-28T15:01:33.460326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461116+0000","last_change":"2024-03-28T15:01:31.095439+0000","last_active":"2024-03-28T15:01:33.461116+0000","last_peered":"2024-03-28T15:01:33.461116+0000","last_clean":"2024-03-28T15:01:33.461116+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:01:33.461116+0000","last_undegraded":"2024-03-28T15:01:33.461116+0000","last_fullsized":"2024-03-28T15:01:33.461116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460849+0000","last_change":"2024-03-28T15:01:32.122655+0000","last_active":"2024-03-28T15:01:33.460849+0000","last_peered":"2024-03-28T15:01:33.460849+0000","last_clean":"2024-03-28T15:01:33.460849+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:01:33.460849+0000","last_undegraded":"2024-03-28T15:01:33.460849+0000","last_fullsized":"2024-03-28T15:01:33.460849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:05.942532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460385+0000","last_change":"2024-03-28T15:01:31.093265+0000","last_active":"2024-03-28T15:01:33.460385+0000","last_peered":"2024-03-28T15:01:33.460385+0000","last_clean":"2024-03-28T15:01:33.460385+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:01:33.460385+0000","last_undegraded":"2024-03-28T15:01:33.460385+0000","last_fullsized":"2024-03-28T15:01:33.460385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104892+0000","last_change":"2024-03-28T15:01:32.108597+0000","last_active":"2024-03-28T15:01:33.104892+0000","last_peered":"2024-03-28T15:01:33.104892+0000","last_clean":"2024-03-28T15:01:33.104892+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:01:33.104892+0000","last_undegraded":"2024-03-28T15:01:33.104892+0000","last_fullsized":"2024-03-28T15:01:33.104892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108209+0000","last_change":"2024-03-28T15:01:31.094750+0000","last_active":"2024-03-28T15:01:33.108209+0000","last_peered":"2024-03-28T15:01:33.108209+0000","last_clean":"2024-03-28T15:01:33.108209+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:01:33.108209+0000","last_undegraded":"2024-03-28T15:01:33.108209+0000","last_fullsized":"2024-03-28T15:01:33.108209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106970+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.106970+0000","last_peered":"2024-03-28T15:01:33.106970+0000","last_clean":"2024-03-28T15:01:33.106970+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:01:33.106970+0000","last_undegraded":"2024-03-28T15:01:33.106970+0000","last_fullsized":"2024-03-28T15:01:33.106970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:24.747778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104874+0000","last_change":"2024-03-28T15:01:31.093706+0000","last_active":"2024-03-28T15:01:33.104874+0000","last_peered":"2024-03-28T15:01:33.104874+0000","last_clean":"2024-03-28T15:01:33.104874+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:01:33.104874+0000","last_undegraded":"2024-03-28T15:01:33.104874+0000","last_fullsized":"2024-03-28T15:01:33.104874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108112+0000","last_change":"2024-03-28T15:01:32.122840+0000","last_active":"2024-03-28T15:01:33.108112+0000","last_peered":"2024-03-28T15:01:33.108112+0000","last_clean":"2024-03-28T15:01:33.108112+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:01:33.108112+0000","last_undegraded":"2024-03-28T15:01:33.108112+0000","last_fullsized":"2024-03-28T15:01:33.108112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461371+0000","last_change":"2024-03-28T15:01:31.095126+0000","last_active":"2024-03-28T15:01:33.461371+0000","last_peered":"2024-03-28T15:01:33.461371+0000","last_clean":"2024-03-28T15:01:33.461371+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:01:33.461371+0000","last_undegraded":"2024-03-28T15:01:33.461371+0000","last_fullsized":"2024-03-28T15:01:33.461371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:21.669535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104885+0000","last_change":"2024-03-28T15:01:32.111925+0000","last_active":"2024-03-28T15:01:33.104885+0000","last_peered":"2024-03-28T15:01:33.104885+0000","last_clean":"2024-03-28T15:01:33.104885+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:01:33.104885+0000","last_undegraded":"2024-03-28T15:01:33.104885+0000","last_fullsized":"2024-03-28T15:01:33.104885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104964+0000","last_change":"2024-03-28T15:01:31.093197+0000","last_active":"2024-03-28T15:01:33.104964+0000","last_peered":"2024-03-28T15:01:33.104964+0000","last_clean":"2024-03-28T15:01:33.104964+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:01:33.104964+0000","last_undegraded":"2024-03-28T15:01:33.104964+0000","last_fullsized":"2024-03-28T15:01:33.104964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461153+0000","last_change":"2024-03-28T15:01:32.115432+0000","last_active":"2024-03-28T15:01:33.461153+0000","last_peered":"2024-03-28T15:01:33.461153+0000","last_clean":"2024-03-28T15:01:33.461153+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:01:33.461153+0000","last_undegraded":"2024-03-28T15:01:33.461153+0000","last_fullsized":"2024-03-28T15:01:33.461153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460639+0000","last_change":"2024-03-28T15:01:31.088204+0000","last_active":"2024-03-28T15:01:33.460639+0000","last_peered":"2024-03-28T15:01:33.460639+0000","last_clean":"2024-03-28T15:01:33.460639+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:01:33.460639+0000","last_undegraded":"2024-03-28T15:01:33.460639+0000","last_fullsized":"2024-03-28T15:01:33.460639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461225+0000","last_change":"2024-03-28T15:01:32.114331+0000","last_active":"2024-03-28T15:01:33.461225+0000","last_peered":"2024-03-28T15:01:33.461225+0000","last_clean":"2024-03-28T15:01:33.461225+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:01:33.461225+0000","last_undegraded":"2024-03-28T15:01:33.461225+0000","last_fullsized":"2024-03-28T15:01:33.461225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460969+0000","last_change":"2024-03-28T15:01:31.095371+0000","last_active":"2024-03-28T15:01:33.460969+0000","last_peered":"2024-03-28T15:01:33.460969+0000","last_clean":"2024-03-28T15:01:33.460969+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:01:33.460969+0000","last_undegraded":"2024-03-28T15:01:33.460969+0000","last_fullsized":"2024-03-28T15:01:33.460969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107409+0000","last_change":"2024-03-28T15:01:32.110950+0000","last_active":"2024-03-28T15:01:33.107409+0000","last_peered":"2024-03-28T15:01:33.107409+0000","last_clean":"2024-03-28T15:01:33.107409+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:01:33.107409+0000","last_undegraded":"2024-03-28T15:01:33.107409+0000","last_fullsized":"2024-03-28T15:01:33.107409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105009+0000","last_change":"2024-03-28T15:01:31.094542+0000","last_active":"2024-03-28T15:01:33.105009+0000","last_peered":"2024-03-28T15:01:33.105009+0000","last_clean":"2024-03-28T15:01:33.105009+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:01:33.105009+0000","last_undegraded":"2024-03-28T15:01:33.105009+0000","last_fullsized":"2024-03-28T15:01:33.105009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460466+0000","last_change":"2024-03-28T15:01:32.112158+0000","last_active":"2024-03-28T15:01:33.460466+0000","last_peered":"2024-03-28T15:01:33.460466+0000","last_clean":"2024-03-28T15:01:33.460466+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:01:33.460466+0000","last_undegraded":"2024-03-28T15:01:33.460466+0000","last_fullsized":"2024-03-28T15:01:33.460466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104884+0000","last_change":"2024-03-28T15:01:31.094440+0000","last_active":"2024-03-28T15:01:33.104884+0000","last_peered":"2024-03-28T15:01:33.104884+0000","last_clean":"2024-03-28T15:01:33.104884+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:01:33.104884+0000","last_undegraded":"2024-03-28T15:01:33.104884+0000","last_fullsized":"2024-03-28T15:01:33.104884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460758+0000","last_change":"2024-03-28T15:01:32.115192+0000","last_active":"2024-03-28T15:01:33.460758+0000","last_peered":"2024-03-28T15:01:33.460758+0000","last_clean":"2024-03-28T15:01:33.460758+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:01:33.460758+0000","last_undegraded":"2024-03-28T15:01:33.460758+0000","last_fullsized":"2024-03-28T15:01:33.460758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461053+0000","last_change":"2024-03-28T15:01:31.096352+0000","last_active":"2024-03-28T15:01:33.461053+0000","last_peered":"2024-03-28T15:01:33.461053+0000","last_clean":"2024-03-28T15:01:33.461053+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:01:33.461053+0000","last_undegraded":"2024-03-28T15:01:33.461053+0000","last_fullsized":"2024-03-28T15:01:33.461053+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460555+0000","last_change":"2024-03-28T15:01:32.124700+0000","last_active":"2024-03-28T15:01:33.460555+0000","last_peered":"2024-03-28T15:01:33.460555+0000","last_clean":"2024-03-28T15:01:33.460555+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:01:33.460555+0000","last_undegraded":"2024-03-28T15:01:33.460555+0000","last_fullsized":"2024-03-28T15:01:33.460555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107999+0000","last_change":"2024-03-28T15:01:32.123638+0000","last_active":"2024-03-28T15:01:33.107999+0000","last_peered":"2024-03-28T15:01:33.107999+0000","last_clean":"2024-03-28T15:01:33.107999+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:01:33.107999+0000","last_undegraded":"2024-03-28T15:01:33.107999+0000","last_fullsized":"2024-03-28T15:01:33.107999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460734+0000","last_change":"2024-03-28T15:01:31.094281+0000","last_active":"2024-03-28T15:01:33.460734+0000","last_peered":"2024-03-28T15:01:33.460734+0000","last_clean":"2024-03-28T15:01:33.460734+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:01:33.460734+0000","last_undegraded":"2024-03-28T15:01:33.460734+0000","last_fullsized":"2024-03-28T15:01:33.460734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107988+0000","last_change":"2024-03-28T15:01:32.112496+0000","last_active":"2024-03-28T15:01:33.107988+0000","last_peered":"2024-03-28T15:01:33.107988+0000","last_clean":"2024-03-28T15:01:33.107988+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:01:33.107988+0000","last_undegraded":"2024-03-28T15:01:33.107988+0000","last_fullsized":"2024-03-28T15:01:33.107988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461219+0000","last_change":"2024-03-28T15:01:31.093773+0000","last_active":"2024-03-28T15:01:33.461219+0000","last_peered":"2024-03-28T15:01:33.461219+0000","last_clean":"2024-03-28T15:01:33.461219+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:01:33.461219+0000","last_undegraded":"2024-03-28T15:01:33.461219+0000","last_fullsized":"2024-03-28T15:01:33.461219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:54.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460976+0000","last_change":"2024-03-28T15:01:32.117275+0000","last_active":"2024-03-28T15:01:33.460976+0000","last_peered":"2024-03-28T15:01:33.460976+0000","last_clean":"2024-03-28T15:01:33.460976+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:01:33.460976+0000","last_undegraded":"2024-03-28T15:01:33.460976+0000","last_fullsized":"2024-03-28T15:01:33.460976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461003+0000","last_change":"2024-03-28T15:01:31.096725+0000","last_active":"2024-03-28T15:01:33.461003+0000","last_peered":"2024-03-28T15:01:33.461003+0000","last_clean":"2024-03-28T15:01:33.461003+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:01:33.461003+0000","last_undegraded":"2024-03-28T15:01:33.461003+0000","last_fullsized":"2024-03-28T15:01:33.461003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460734+0000","last_change":"2024-03-28T15:01:32.120071+0000","last_active":"2024-03-28T15:01:33.460734+0000","last_peered":"2024-03-28T15:01:33.460734+0000","last_clean":"2024-03-28T15:01:33.460734+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:01:33.460734+0000","last_undegraded":"2024-03-28T15:01:33.460734+0000","last_fullsized":"2024-03-28T15:01:33.460734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461240+0000","last_change":"2024-03-28T15:01:31.095351+0000","last_active":"2024-03-28T15:01:33.461240+0000","last_peered":"2024-03-28T15:01:33.461240+0000","last_clean":"2024-03-28T15:01:33.461240+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:01:33.461240+0000","last_undegraded":"2024-03-28T15:01:33.461240+0000","last_fullsized":"2024-03-28T15:01:33.461240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":661302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":661302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:01:36.432 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T15:01:36.432 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T15:01:36.433 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-28T15:01:36.433 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-28T15:01:36.433 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-28T15:01:36.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.570+0000 7f3ce9e81640 1 Processor -- start 2024-03-28T15:01:36.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.570+0000 7f3ce9e81640 1 -- start start 2024-03-28T15:01:36.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce9e81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce4159460 0x7f3ce4152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce9e81640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce406fd00 con 0x7f3ce41527b0 2024-03-28T15:01:36.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce9e81640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce406ffe0 con 0x7f3ce40c4700 2024-03-28T15:01:36.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce9e81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce4062f00 con 0x7f3ce4159460 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce2ffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f3ce41527b0 0x7f3ce4158d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:37424/0 (socket says 172.21.15.186:37424) 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce2ffd640 1 -- 172.21.15.186:0/1951233823 learned_addr learned my addr 172.21.15.186:0/1951233823 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce3fff640 1 --2- 172.21.15.186:0/1951233823 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce4159460 0x7f3ce4152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce27fc640 1 -- 172.21.15.186:0/1951233823 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2832293237 0 0) 0x7f3ce406fd00 con 0x7f3ce41527b0 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce27fc640 1 -- 172.21.15.186:0/1951233823 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ccc003520 con 0x7f3ce41527b0 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce27fc640 1 -- 172.21.15.186:0/1951233823 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3316015921 0 0) 0x7f3ce406ffe0 con 0x7f3ce40c4700 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce27fc640 1 -- 172.21.15.186:0/1951233823 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ce406fd00 con 0x7f3ce40c4700 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce27fc640 1 -- 172.21.15.186:0/1951233823 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4247187256 0 0) 0x7f3ccc003520 con 0x7f3ce41527b0 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce3fff640 1 -- 172.21.15.186:0/1951233823 >> v1:172.21.15.191:6789/0 conn(0x7f3ce40c4700 legacy=0x7f3ce40c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce3fff640 1 -- 172.21.15.186:0/1951233823 >> v1:172.21.15.186:6789/0 conn(0x7f3ce41527b0 legacy=0x7f3ce4158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.571+0000 7f3ce3fff640 1 -- 172.21.15.186:0/1951233823 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce4062940 con 0x7f3ce4159460 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.572+0000 7f3ce3fff640 1 --2- 172.21.15.186:0/1951233823 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce4159460 0x7f3ce4152c40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009dc0 tx=0x7f3cdc02be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9ece806593a2777 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.572+0000 7f3ce27fc640 1 -- 172.21.15.186:0/1951233823 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc039070 con 0x7f3ce4159460 2024-03-28T15:01:36.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.572+0000 7f3ce27fc640 1 -- 172.21.15.186:0/1951233823 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cdc02c9a0 con 0x7f3ce4159460 2024-03-28T15:01:36.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.572+0000 7f3ce27fc640 1 -- 172.21.15.186:0/1951233823 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8001670 con 0x7f3ce4159460 2024-03-28T15:01:36.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.573+0000 7f3ce9e81640 1 -- 172.21.15.186:0/1951233823 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce4159460 msgr2=0x7f3ce4152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.573+0000 7f3ce9e81640 1 --2- 172.21.15.186:0/1951233823 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce4159460 0x7f3ce4152c40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009dc0 tx=0x7f3cdc02be70 comp rx=0 tx=0).stop 2024-03-28T15:01:36.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.573+0000 7f3ce9e81640 1 -- 172.21.15.186:0/1951233823 shutdown_connections 2024-03-28T15:01:36.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.573+0000 7f3ce9e81640 1 --2- 172.21.15.186:0/1951233823 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce4159460 0x7f3ce4152c40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.573+0000 7f3ce9e81640 1 -- 172.21.15.186:0/1951233823 >> 172.21.15.186:0/1951233823 conn(0x7f3ce40cc580 msgr2=0x7f3ce40cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:36.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.573+0000 7f3ce9e81640 1 -- 172.21.15.186:0/1951233823 shutdown_connections 2024-03-28T15:01:36.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.573+0000 7f3ce9e81640 1 -- 172.21.15.186:0/1951233823 wait complete. 2024-03-28T15:01:36.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.574+0000 7f3ce9e81640 1 Processor -- start 2024-03-28T15:01:36.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.574+0000 7f3ce9e81640 1 -- start start 2024-03-28T15:01:36.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce9e81640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ce40c4700 0x7f3ce40cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce9e81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce41527b0 0x7f3ce41eea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce9e81640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce4159460 0x7f3ce4147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce9e81640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce40c66b0 con 0x7f3ce4159460 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce9e81640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce40c8df0 con 0x7f3ce40c4700 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce9e81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce40c6830 con 0x7f3ce41527b0 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce4159460 0x7f3ce4147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce4159460 0x7f3ce4147020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41212/0 (socket says 172.21.15.186:41212) 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce3fff640 1 -- 172.21.15.186:0/2516167084 learned_addr learned my addr 172.21.15.186:0/2516167084 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce37fe640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ce40c4700 0x7f3ce40cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce2ffd640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce41527b0 0x7f3ce41eea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce3fff640 1 -- 172.21.15.186:0/2516167084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce41527b0 msgr2=0x7f3ce41eea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce3fff640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce41527b0 0x7f3ce41eea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce3fff640 1 -- 172.21.15.186:0/2516167084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ce40c4700 msgr2=0x7f3ce40cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce3fff640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ce40c4700 0x7f3ce40cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce3fff640 1 -- 172.21.15.186:0/2516167084 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce41e3cd0 con 0x7f3ce4159460 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce37fe640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ce40c4700 0x7f3ce40cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce2ffd640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce41527b0 0x7f3ce41eea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.575+0000 7f3ce3fff640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce4159460 0x7f3ce4147020 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009ee0 tx=0x7f3cdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.576+0000 7f3ccbfff640 1 -- 172.21.15.186:0/2516167084 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc039050 con 0x7f3ce4159460 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.576+0000 7f3ccbfff640 1 -- 172.21.15.186:0/2516167084 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cdc045070 con 0x7f3ce4159460 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.576+0000 7f3ce9e81640 1 -- 172.21.15.186:0/2516167084 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce41ee2d0 con 0x7f3ce4159460 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.576+0000 7f3ccbfff640 1 -- 172.21.15.186:0/2516167084 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc009450 con 0x7f3ce4159460 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.576+0000 7f3ce9e81640 1 -- 172.21.15.186:0/2516167084 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce41e29e0 con 0x7f3ce4159460 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.577+0000 7f3ce9e81640 1 -- 172.21.15.186:0/2516167084 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce4069500 con 0x7f3ce4159460 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.577+0000 7f3ccbfff640 1 -- 172.21.15.186:0/2516167084 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3cdc040690 con 0x7f3ce4159460 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.577+0000 7f3ccbfff640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3cd006c530 0x7f3cd006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.577+0000 7f3ccbfff640 1 -- 172.21.15.186:0/2516167084 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3cdc0ae070 con 0x7f3ce4159460 2024-03-28T15:01:36.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.577+0000 7f3ce37fe640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3cd006c530 0x7f3cd006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.578+0000 7f3ce37fe640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3cd006c530 0x7f3cd006e9f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4001710 tx=0x7f3cd4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:36.581 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.582+0000 7f3ccbfff640 1 -- 172.21.15.186:0/2516167084 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3cdc07c690 con 0x7f3ce4159460 2024-03-28T15:01:36.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.807+0000 7f3ce9e81640 1 -- 172.21.15.186:0/2516167084 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3ce414b730 con 0x7f3ce4159460 2024-03-28T15:01:36.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.810+0000 7f3ccbfff640 1 -- 172.21.15.186:0/2516167084 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+173740 (secure 0 0 0) 0x7f3cdc07bf90 con 0x7f3ce4159460 2024-03-28T15:01:36.813 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:36.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 -- 172.21.15.186:0/2516167084 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3cd006c530 msgr2=0x7f3cd006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3cd006c530 0x7f3cd006e9f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4001710 tx=0x7f3cd4009040 comp rx=0 tx=0).stop 2024-03-28T15:01:36.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 -- 172.21.15.186:0/2516167084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce4159460 msgr2=0x7f3ce4147020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce4159460 0x7f3ce4147020 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009ee0 tx=0x7f3cdc004a40 comp rx=0 tx=0).stop 2024-03-28T15:01:36.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 -- 172.21.15.186:0/2516167084 shutdown_connections 2024-03-28T15:01:36.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3cd006c530 0x7f3cd006e9f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce4159460 0x7f3ce4147020 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce41527b0 0x7f3ce41eea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 --2- 172.21.15.186:0/2516167084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ce40c4700 0x7f3ce40cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 -- 172.21.15.186:0/2516167084 >> 172.21.15.186:0/2516167084 conn(0x7f3ce40cc580 msgr2=0x7f3ce4151940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:36.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 -- 172.21.15.186:0/2516167084 shutdown_connections 2024-03-28T15:01:36.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.813+0000 7f3ce9e81640 1 -- 172.21.15.186:0/2516167084 wait complete. 2024-03-28T15:01:36.824 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2995751804},{"type":"v1","addr":"172.21.15.186:6833","nonce":2995751804}]},"active_addr":"172.21.15.186:6833/2995751804","active_change":"2024-03-28T15:01:10.626661+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":1608731397}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":2563282825}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":809630734}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":921484030}]}]} 2024-03-28T15:01:36.827 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-28T15:01:36.828 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-28T15:01:36.828 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:36.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.928+0000 7f39e22fb640 1 Processor -- start 2024-03-28T15:01:36.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.928+0000 7f39e22fb640 1 -- start start 2024-03-28T15:01:36.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39e22fb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc151990 0x7f39dc0c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39e22fb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39dc06caa0 con 0x7f39dc0c2d00 2024-03-28T15:01:36.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39e22fb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39dc06cd80 con 0x7f39dc0b1e30 2024-03-28T15:01:36.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39e22fb640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f39dc0c1da0 con 0x7f39dc151990 2024-03-28T15:01:36.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39e0898640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f39dc0c2d00 0x7f39dc161dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:37426/0 (socket says 172.21.15.186:37426) 2024-03-28T15:01:36.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39e0898640 1 -- 172.21.15.186:0/3406055763 learned_addr learned my addr 172.21.15.186:0/3406055763 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:36.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1128875854 0 0) 0x7f39dc06caa0 con 0x7f39dc0c2d00 2024-03-28T15:01:36.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39dbfff640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc151990 0x7f39dc0c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39c4003520 con 0x7f39dc0c2d00 2024-03-28T15:01:36.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3263629168 0 0) 0x7f39c4003520 con 0x7f39dc0c2d00 2024-03-28T15:01:36.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc151990 msgr2=0x7f39dc0c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc151990 0x7f39dc0c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 >> v1:172.21.15.191:6789/0 conn(0x7f39dc0b1e30 legacy=0x7f39dc0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39dc0c1340 con 0x7f39dc0c2d00 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f39d40019f0 con 0x7f39dc0c2d00 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 >> v1:172.21.15.186:6789/0 conn(0x7f39dc0c2d00 legacy=0x7f39dc161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39c40039e0 0x7f39c4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39c4004800 0x7f39c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc151990 0x7f39c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f39dc0c1da0 con 0x7f39c4004800 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f39c400bb00 con 0x7f39c40039e0 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.929+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f39c400bd00 con 0x7f39dc151990 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.930+0000 7f39db7fe640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39c4004800 0x7f39c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.930+0000 7f39dbfff640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39c40039e0 0x7f39c4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.930+0000 7f39e0898640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc151990 0x7f39c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.930+0000 7f39db7fe640 1 -- 172.21.15.186:0/3406055763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc151990 msgr2=0x7f39c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:36.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.930+0000 7f39db7fe640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc151990 0x7f39c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.930+0000 7f39db7fe640 1 -- 172.21.15.186:0/3406055763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39c40039e0 msgr2=0x7f39c4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:36.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.930+0000 7f39db7fe640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39c40039e0 0x7f39c4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.930+0000 7f39db7fe640 1 -- 172.21.15.186:0/3406055763 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c4003520 con 0x7f39c4004800 2024-03-28T15:01:36.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.931+0000 7f39db7fe640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39c4004800 0x7f39c4004c80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0b2250 tx=0x7f39c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10fcaa30b4205698 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:36.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.931+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0001370 con 0x7f39c4004800 2024-03-28T15:01:36.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.931+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d00052d0 con 0x7f39c4004800 2024-03-28T15:01:36.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.931+0000 7f39daffd640 1 -- 172.21.15.186:0/3406055763 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0003830 con 0x7f39c4004800 2024-03-28T15:01:36.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.931+0000 7f39e22fb640 1 -- 172.21.15.186:0/3406055763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39c4004800 msgr2=0x7f39c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.931+0000 7f39e22fb640 1 --2- 172.21.15.186:0/3406055763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39c4004800 0x7f39c4004c80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0b2250 tx=0x7f39c4005830 comp rx=0 tx=0).stop 2024-03-28T15:01:36.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.931+0000 7f39e0898640 1 -- 172.21.15.186:0/3406055763 reap_dead start 2024-03-28T15:01:36.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.931+0000 7f39e22fb640 1 -- 172.21.15.186:0/3406055763 shutdown_connections 2024-03-28T15:01:36.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.931+0000 7f39e22fb640 1 -- 172.21.15.186:0/3406055763 >> 172.21.15.186:0/3406055763 conn(0x7f39dc0c8a20 msgr2=0x7f39dc0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:36.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.931+0000 7f39e22fb640 1 -- 172.21.15.186:0/3406055763 shutdown_connections 2024-03-28T15:01:36.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.931+0000 7f39e22fb640 1 -- 172.21.15.186:0/3406055763 wait complete. 2024-03-28T15:01:36.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.932+0000 7f39e22fb640 1 Processor -- start 2024-03-28T15:01:36.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39e22fb640 1 -- start start 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39e22fb640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39c4004800 0x7f39dc1e3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39e22fb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39dc150360 0x7f39dc156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39e22fb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc157000 0x7f39dc0caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39e22fb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f39dc073dd0 con 0x7f39dc150360 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39e22fb640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f39dc0c1900 con 0x7f39c4004800 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39e22fb640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f39dc073f50 con 0x7f39dc157000 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39db7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39dc150360 0x7f39dc156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39db7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39dc150360 0x7f39dc156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41222/0 (socket says 172.21.15.186:41222) 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39db7fe640 1 -- 172.21.15.186:0/2562900283 learned_addr learned my addr 172.21.15.186:0/2562900283 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39e0898640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc157000 0x7f39dc0caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.933+0000 7f39dbfff640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39c4004800 0x7f39dc1e3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.934+0000 7f39db7fe640 1 -- 172.21.15.186:0/2562900283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc157000 msgr2=0x7f39dc0caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.934+0000 7f39db7fe640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc157000 0x7f39dc0caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.934+0000 7f39db7fe640 1 -- 172.21.15.186:0/2562900283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39c4004800 msgr2=0x7f39dc1e3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.934+0000 7f39db7fe640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39c4004800 0x7f39dc1e3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.934+0000 7f39db7fe640 1 -- 172.21.15.186:0/2562900283 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39dc1508c0 con 0x7f39dc150360 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.934+0000 7f39db7fe640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39dc150360 0x7f39dc156ac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f39d0005040 tx=0x7f39d0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.934+0000 7f39d8ff9640 1 -- 172.21.15.186:0/2562900283 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0041020 con 0x7f39dc150360 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.934+0000 7f39d8ff9640 1 -- 172.21.15.186:0/2562900283 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d0005480 con 0x7f39dc150360 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.934+0000 7f39e22fb640 1 -- 172.21.15.186:0/2562900283 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39dc150b80 con 0x7f39dc150360 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.934+0000 7f39d8ff9640 1 -- 172.21.15.186:0/2562900283 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0007a30 con 0x7f39dc150360 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.934+0000 7f39e22fb640 1 -- 172.21.15.186:0/2562900283 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39dc0ca4a0 con 0x7f39dc150360 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.935+0000 7f39e22fb640 1 -- 172.21.15.186:0/2562900283 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39dc14ef50 con 0x7f39dc150360 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.935+0000 7f39d8ff9640 1 -- 172.21.15.186:0/2562900283 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f39d0044020 con 0x7f39dc150360 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.936+0000 7f39d8ff9640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f39c806c530 0x7f39c806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.936+0000 7f39d8ff9640 1 -- 172.21.15.186:0/2562900283 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f39d00acfa0 con 0x7f39dc150360 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.936+0000 7f39dbfff640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f39c806c530 0x7f39c806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:36.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.936+0000 7f39dbfff640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f39c806c530 0x7f39c806e9f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39cc009b40 tx=0x7f39cc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:36.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:36.940+0000 7f39d8ff9640 1 -- 172.21.15.186:0/2562900283 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f39dc14ef50 con 0x7f39dc150360 2024-03-28T15:01:37.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.123+0000 7f39e22fb640 1 -- 172.21.15.186:0/2562900283 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39dc069500 con 0x7f39dc150360 2024-03-28T15:01:37.124 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.124+0000 7f39d8ff9640 1 -- 172.21.15.186:0/2562900283 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f39d004b030 con 0x7f39dc150360 2024-03-28T15:01:37.124 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:37.125 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:33.099254+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:37.127 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 -- 172.21.15.186:0/2562900283 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f39c806c530 msgr2=0x7f39c806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.127 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f39c806c530 0x7f39c806e9f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39cc009b40 tx=0x7f39cc009190 comp rx=0 tx=0).stop 2024-03-28T15:01:37.127 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 -- 172.21.15.186:0/2562900283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39dc150360 msgr2=0x7f39dc156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.127 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39dc150360 0x7f39dc156ac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f39d0005040 tx=0x7f39d0001120 comp rx=0 tx=0).stop 2024-03-28T15:01:37.127 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 -- 172.21.15.186:0/2562900283 shutdown_connections 2024-03-28T15:01:37.127 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f39c806c530 0x7f39c806e9f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39dc157000 0x7f39dc0caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39dc150360 0x7f39dc156ac0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 --2- 172.21.15.186:0/2562900283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39c4004800 0x7f39dc1e3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 -- 172.21.15.186:0/2562900283 >> 172.21.15.186:0/2562900283 conn(0x7f39dc0c8a20 msgr2=0x7f39dc0c7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 -- 172.21.15.186:0/2562900283 shutdown_connections 2024-03-28T15:01:37.128 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.127+0000 7f39e22fb640 1 -- 172.21.15.186:0/2562900283 wait complete. 2024-03-28T15:01:37.139 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-28T15:01:37.139 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:37.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.239+0000 7f7a469a2640 1 Processor -- start 2024-03-28T15:01:37.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.240+0000 7f7a469a2640 1 -- start start 2024-03-28T15:01:37.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.240+0000 7f7a469a2640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a400d0be0 0x7f7a4015b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.240+0000 7f7a469a2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7a400c5ec0 0x7f7a400b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.240+0000 7f7a469a2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a4015e4e0 con 0x7f7a400d0800 2024-03-28T15:01:37.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.240+0000 7f7a469a2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f7a40072af0 con 0x7f7a400c5ec0 2024-03-28T15:01:37.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.240+0000 7f7a469a2640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4015ed60 con 0x7f7a400d0be0 2024-03-28T15:01:37.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.240+0000 7f7a461a1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7a400c5ec0 0x7f7a400b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.240+0000 7f7a461a1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7a400c5ec0 0x7f7a400b36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41228/0 (socket says 172.21.15.186:41228) 2024-03-28T15:01:37.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.240+0000 7f7a461a1640 1 -- 172.21.15.186:0/128714963 learned_addr learned my addr 172.21.15.186:0/128714963 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.241+0000 7f7a4519f640 1 --2- 172.21.15.186:0/128714963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a400d0be0 0x7f7a4015b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.241+0000 7f7a461a1640 1 -- 172.21.15.186:0/128714963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a400d0be0 msgr2=0x7f7a4015b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.241+0000 7f7a461a1640 1 --2- 172.21.15.186:0/128714963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a400d0be0 0x7f7a4015b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.241+0000 7f7a461a1640 1 -- 172.21.15.186:0/128714963 >> v1:172.21.15.191:6789/0 conn(0x7f7a400d0800 legacy=0x7f7a4015ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.241+0000 7f7a461a1640 1 -- 172.21.15.186:0/128714963 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4015e7a0 con 0x7f7a400c5ec0 2024-03-28T15:01:37.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.241+0000 7f7a461a1640 1 --2- 172.21.15.186:0/128714963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a400c5ec0 0x7f7a400b36b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009870 tx=0x7f7a3002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66842b6bc46e40f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.242+0000 7f7a4499e640 1 -- 172.21.15.186:0/128714963 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a30034070 con 0x7f7a400c5ec0 2024-03-28T15:01:37.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.242+0000 7f7a4499e640 1 -- 172.21.15.186:0/128714963 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a30030840 con 0x7f7a400c5ec0 2024-03-28T15:01:37.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.242+0000 7f7a4499e640 1 -- 172.21.15.186:0/128714963 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a30030c10 con 0x7f7a400c5ec0 2024-03-28T15:01:37.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.242+0000 7f7a469a2640 1 -- 172.21.15.186:0/128714963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a400c5ec0 msgr2=0x7f7a400b36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.242+0000 7f7a469a2640 1 --2- 172.21.15.186:0/128714963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a400c5ec0 0x7f7a400b36b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009870 tx=0x7f7a3002b980 comp rx=0 tx=0).stop 2024-03-28T15:01:37.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.242+0000 7f7a469a2640 1 -- 172.21.15.186:0/128714963 shutdown_connections 2024-03-28T15:01:37.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.242+0000 7f7a469a2640 1 --2- 172.21.15.186:0/128714963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a400c5ec0 0x7f7a400b36b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.242+0000 7f7a469a2640 1 --2- 172.21.15.186:0/128714963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a400d0be0 0x7f7a4015b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.242+0000 7f7a469a2640 1 -- 172.21.15.186:0/128714963 >> 172.21.15.186:0/128714963 conn(0x7f7a400735c0 msgr2=0x7f7a400739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.242+0000 7f7a469a2640 1 -- 172.21.15.186:0/128714963 shutdown_connections 2024-03-28T15:01:37.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.242+0000 7f7a469a2640 1 -- 172.21.15.186:0/128714963 wait complete. 2024-03-28T15:01:37.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.243+0000 7f7a469a2640 1 Processor -- start 2024-03-28T15:01:37.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.243+0000 7f7a469a2640 1 -- start start 2024-03-28T15:01:37.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a469a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a400c5ec0 0x7f7a4014add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a469a2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a400d0800 0x7f7a400ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a469a2640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a400d0be0 0x7f7a401f4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a469a2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4015ac60 con 0x7f7a400c5ec0 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a469a2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4015a980 con 0x7f7a400d0800 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a469a2640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4015af40 con 0x7f7a400d0be0 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a459a0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a400c5ec0 0x7f7a4014add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a459a0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a400c5ec0 0x7f7a4014add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41238/0 (socket says 172.21.15.186:41238) 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a459a0640 1 -- 172.21.15.186:0/2767008816 learned_addr learned my addr 172.21.15.186:0/2767008816 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a4519f640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a400d0800 0x7f7a400ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a461a1640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a400d0be0 0x7f7a401f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a459a0640 1 -- 172.21.15.186:0/2767008816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a400d0be0 msgr2=0x7f7a401f4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a459a0640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a400d0be0 0x7f7a401f4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a459a0640 1 -- 172.21.15.186:0/2767008816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a400d0800 msgr2=0x7f7a400ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a459a0640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a400d0800 0x7f7a400ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a459a0640 1 -- 172.21.15.186:0/2767008816 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4014b310 con 0x7f7a400c5ec0 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.244+0000 7f7a4519f640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a400d0800 0x7f7a400ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.245+0000 7f7a459a0640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a400c5ec0 0x7f7a4014add0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c7f0 tx=0x7f7a3800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.245+0000 7f7a3e7fc640 1 -- 172.21.15.186:0/2767008816 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38016020 con 0x7f7a400c5ec0 2024-03-28T15:01:37.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.245+0000 7f7a3e7fc640 1 -- 172.21.15.186:0/2767008816 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a3800aa50 con 0x7f7a400c5ec0 2024-03-28T15:01:37.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.245+0000 7f7a3e7fc640 1 -- 172.21.15.186:0/2767008816 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3800ae40 con 0x7f7a400c5ec0 2024-03-28T15:01:37.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.245+0000 7f7a469a2640 1 -- 172.21.15.186:0/2767008816 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a400cae40 con 0x7f7a400c5ec0 2024-03-28T15:01:37.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.245+0000 7f7a469a2640 1 -- 172.21.15.186:0/2767008816 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a400cc3b0 con 0x7f7a400c5ec0 2024-03-28T15:01:37.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.246+0000 7f7a469a2640 1 -- 172.21.15.186:0/2767008816 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a40069500 con 0x7f7a400c5ec0 2024-03-28T15:01:37.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.246+0000 7f7a3e7fc640 1 -- 172.21.15.186:0/2767008816 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7a38019020 con 0x7f7a400c5ec0 2024-03-28T15:01:37.249 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.247+0000 7f7a3e7fc640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7a2c06c530 0x7f7a2c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.249 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.247+0000 7f7a3e7fc640 1 -- 172.21.15.186:0/2767008816 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7a3808b800 con 0x7f7a400c5ec0 2024-03-28T15:01:37.249 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.247+0000 7f7a4519f640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7a2c06c530 0x7f7a2c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.249 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.247+0000 7f7a4519f640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7a2c06c530 0x7f7a2c06e9f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7a34006fd0 tx=0x7f7a34008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.251 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.251+0000 7f7a3e7fc640 1 -- 172.21.15.186:0/2767008816 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7a38020030 con 0x7f7a400c5ec0 2024-03-28T15:01:37.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.426+0000 7f7a469a2640 1 -- 172.21.15.186:0/2767008816 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7a40150230 con 0x7f7a400c5ec0 2024-03-28T15:01:37.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.428+0000 7f7a3e7fc640 1 -- 172.21.15.186:0/2767008816 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f7a4014b310 con 0x7f7a400c5ec0 2024-03-28T15:01:37.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:37.428 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:33.099254+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:37.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 -- 172.21.15.186:0/2767008816 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7a2c06c530 msgr2=0x7f7a2c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7a2c06c530 0x7f7a2c06e9f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7a34006fd0 tx=0x7f7a34008040 comp rx=0 tx=0).stop 2024-03-28T15:01:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 -- 172.21.15.186:0/2767008816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a400c5ec0 msgr2=0x7f7a4014add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a400c5ec0 0x7f7a4014add0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c7f0 tx=0x7f7a3800ccb0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 -- 172.21.15.186:0/2767008816 shutdown_connections 2024-03-28T15:01:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7a2c06c530 0x7f7a2c06e9f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a400d0be0 0x7f7a401f4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a400d0800 0x7f7a400ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 --2- 172.21.15.186:0/2767008816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a400c5ec0 0x7f7a4014add0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 -- 172.21.15.186:0/2767008816 >> 172.21.15.186:0/2767008816 conn(0x7f7a400735c0 msgr2=0x7f7a40153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 -- 172.21.15.186:0/2767008816 shutdown_connections 2024-03-28T15:01:37.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.430+0000 7f7a469a2640 1 -- 172.21.15.186:0/2767008816 wait complete. 2024-03-28T15:01:37.441 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-28T15:01:37.442 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-28T15:01:37.442 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-28T15:01:37.442 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-28T15:01:37.443 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-28T15:01:37.443 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-28T15:01:37.444 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-28T15:01:37.444 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-28T15:01:37.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.613+0000 7ff4c1c21640 1 Processor -- start 2024-03-28T15:01:37.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.613+0000 7ff4c1c21640 1 -- start start 2024-03-28T15:01:37.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.614+0000 7ff4c1c21640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff4bc073220 0x7ff4bc0c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.614+0000 7ff4c1c21640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4bc16b620 0x7ff4bc06cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.614+0000 7ff4c1c21640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff4bc0c7fc0 0x7ff4bc0c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.614+0000 7ff4c1c21640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff4bc16a8f0 con 0x7ff4bc073220 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.614+0000 7ff4c1c21640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7ff4bc0c70a0 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.614+0000 7ff4c1c21640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff4bc073a40 con 0x7ff4bc16b620 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.615+0000 7ff4c0c1f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff4bc073220 0x7ff4bc0c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.615+0000 7ff4c0c1f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff4bc073220 0x7ff4bc0c6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41248/0 (socket says 172.21.15.186:41248) 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.615+0000 7ff4c0c1f640 1 -- 172.21.15.186:0/2378677303 learned_addr learned my addr 172.21.15.186:0/2378677303 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.615+0000 7ff4c0c1f640 1 -- 172.21.15.186:0/2378677303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4bc16b620 msgr2=0x7ff4bc06cde0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.615+0000 7ff4c0c1f640 1 --2- 172.21.15.186:0/2378677303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4bc16b620 0x7ff4bc06cde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.615+0000 7ff4c0c1f640 1 -- 172.21.15.186:0/2378677303 >> v2:172.21.15.191:3300/0 conn(0x7ff4bc0c7fc0 msgr2=0x7ff4bc0c7480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.615+0000 7ff4c0c1f640 1 --2- 172.21.15.186:0/2378677303 >> v2:172.21.15.191:3300/0 conn(0x7ff4bc0c7fc0 0x7ff4bc0c7480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.615+0000 7ff4c0c1f640 1 -- 172.21.15.186:0/2378677303 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4bc073d20 con 0x7ff4bc073220 2024-03-28T15:01:37.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.616+0000 7f210a66f640 1 Processor -- start 2024-03-28T15:01:37.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.616+0000 7f210a66f640 1 -- start start 2024-03-28T15:01:37.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.616+0000 7fd7c504e640 1 Processor -- start 2024-03-28T15:01:37.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.616+0000 7fd7c504e640 1 -- start start 2024-03-28T15:01:37.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7f210a66f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20fc0dd440 0x7f20fc04b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7f210a66f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f20fc0452c0 0x7f20fc0456e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7f210a66f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f20fc0e6760 0x7f20fc0e6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7fd92d9f7640 1 Processor -- start 2024-03-28T15:01:37.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7f210a66f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f20fc007640 con 0x7f20fc0e6760 2024-03-28T15:01:37.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7f210a66f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f20fc04b830 con 0x7f20fc0452c0 2024-03-28T15:01:37.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7fd92d9f7640 1 -- start start 2024-03-28T15:01:37.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7f210a66f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f20fc006a30 con 0x7f20fc0dd440 2024-03-28T15:01:37.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7f21037fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f20fc0452c0 0x7f20fc0456e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7fabc90f4640 1 Processor -- start 2024-03-28T15:01:37.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7f2103fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20fc0dd440 0x7f20fc04b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7f21037fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f20fc0452c0 0x7f20fc0456e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58220/0 (socket says 172.21.15.186:58220) 2024-03-28T15:01:37.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7f21037fe640 1 -- 172.21.15.186:0/2495608779 learned_addr learned my addr 172.21.15.186:0/2495608779 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.616+0000 7ff4c0c1f640 1 --2- 172.21.15.186:0/2378677303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4bc073220 0x7ff4bc0c6b60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc06e000 tx=0x7ff4b4030270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40e2d68ad5955c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7ff4bb7fe640 1 -- 172.21.15.186:0/2378677303 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4036070 con 0x7ff4bc073220 2024-03-28T15:01:37.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7ff4bb7fe640 1 -- 172.21.15.186:0/2378677303 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b403a600 con 0x7ff4bc073220 2024-03-28T15:01:37.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7ff4bb7fe640 1 -- 172.21.15.186:0/2378677303 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b403a9f0 con 0x7ff4bc073220 2024-03-28T15:01:37.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.617+0000 7f2108c0c640 1 --2- 172.21.15.186:0/2495608779 >> v2:172.21.15.186:3300/0 conn(0x7f20fc0e6760 0x7f20fc0e6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7fd7c504e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd7c0152960 0x7fd7c006e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7fd7c504e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd7c00c8a20 0x7fd7c006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7fd7c504e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7c015c7f0 con 0x7fd7c00cb6d0 2024-03-28T15:01:37.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7f6450c81640 1 Processor -- start 2024-03-28T15:01:37.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7fd7be59c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd7c00cb6d0 0x7fd7c0071170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:36112/0 (socket says 172.21.15.186:36112) 2024-03-28T15:01:37.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7fd7be59c640 1 -- 172.21.15.186:0/1912442024 learned_addr learned my addr 172.21.15.186:0/1912442024 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:37.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7fd7c504e640 1 -- 172.21.15.186:0/1912442024 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c015cad0 con 0x7fd7c0152960 2024-03-28T15:01:37.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7fabc90f4640 1 -- start start 2024-03-28T15:01:37.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7f6450c81640 1 -- start start 2024-03-28T15:01:37.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7ff4c1c21640 1 -- 172.21.15.186:0/2378677303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4bc073220 msgr2=0x7ff4bc0c6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7ff4c1c21640 1 --2- 172.21.15.186:0/2378677303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4bc073220 0x7ff4bc0c6b60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc06e000 tx=0x7ff4b4030270 comp rx=0 tx=0).stop 2024-03-28T15:01:37.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7f21037fe640 1 -- 172.21.15.186:0/2495608779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20fc0dd440 msgr2=0x7f20fc04b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7f21037fe640 1 --2- 172.21.15.186:0/2495608779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20fc0dd440 0x7f20fc04b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7f21037fe640 1 -- 172.21.15.186:0/2495608779 >> v2:172.21.15.186:3300/0 conn(0x7f20fc0e6760 msgr2=0x7f20fc0e6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7f21037fe640 1 --2- 172.21.15.186:0/2495608779 >> v2:172.21.15.186:3300/0 conn(0x7f20fc0e6760 0x7f20fc0e6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7f21037fe640 1 -- 172.21.15.186:0/2495608779 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20fc006d10 con 0x7f20fc0452c0 2024-03-28T15:01:37.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7f2103fff640 1 --2- 172.21.15.186:0/2495608779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20fc0dd440 0x7f20fc04b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:37.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fabc90f4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fabbc0d5e50 0x7fabbc0dcf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fabc90f4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabbc04c690 0x7fabbc0dd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fabc90f4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fabbc0defb0 0x7fabbc0df400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fd92d9f7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd92806ee50 0x7fd9280c8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fabc90f4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fabbc009300 con 0x7fabbc0defb0 2024-03-28T15:01:37.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fabc90f4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fabbc04cc40 con 0x7fabbc0d5e50 2024-03-28T15:01:37.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fabc90f4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fabbc04e9d0 con 0x7fabbc04c690 2024-03-28T15:01:37.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fd92d9f7640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd9281567b0 0x7fd9281505f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fd92d9f7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd928156d10 con 0x7fd92806f230 2024-03-28T15:01:37.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fd92d9f7640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fd92806c940 con 0x7fd9281567b0 2024-03-28T15:01:37.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fd92d9f7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd92806cc20 con 0x7fd92806ee50 2024-03-28T15:01:37.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fabc259c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabbc04c690 0x7fabbc0dd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fabc2d9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fabbc0d5e50 0x7fabbc0dcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fabc259c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabbc04c690 0x7fabbc0dd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:56866/0 (socket says 172.21.15.186:56866) 2024-03-28T15:01:37.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fabc359e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fabbc0defb0 0x7fabbc0df400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fabc259c640 1 -- 172.21.15.186:0/1838894111 learned_addr learned my addr 172.21.15.186:0/1838894111 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fd9267fc640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd92806f230 0x7fd92806d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:37428/0 (socket says 172.21.15.186:37428) 2024-03-28T15:01:37.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fd926ffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd92806ee50 0x7fd9280c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7f890f238640 1 Processor -- start 2024-03-28T15:01:37.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7f890f238640 1 -- start start 2024-03-28T15:01:37.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fd9267fc640 1 -- 172.21.15.186:0/2993624802 learned_addr learned my addr 172.21.15.186:0/2993624802 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:37.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7f8bbba15640 1 Processor -- start 2024-03-28T15:01:37.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fd9277fe640 1 --2- 172.21.15.186:0/2993624802 >> v2:172.21.15.191:3300/0 conn(0x7fd9281567b0 0x7fd9281505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7f21037fe640 1 --2- 172.21.15.186:0/2495608779 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20fc0452c0 0x7f20fc0456e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f20fc008520 tx=0x7f20f802b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eb1780b4fbc4f9d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7fd7c504e640 1 -- 172.21.15.186:0/1912442024 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c00c4cb0 con 0x7fd7c00c8a20 2024-03-28T15:01:37.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.618+0000 7fd7bed9d640 1 --2- 172.21.15.186:0/1912442024 >> v2:172.21.15.186:3300/0 conn(0x7fd7c0152960 0x7fd7c006e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7fd7bf59e640 1 --2- 172.21.15.186:0/1912442024 >> v2:172.21.15.191:3300/0 conn(0x7fd7c00c8a20 0x7fd7c006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7ff4c1c21640 1 -- 172.21.15.186:0/2378677303 shutdown_connections 2024-03-28T15:01:37.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7ff4c1c21640 1 --2- 172.21.15.186:0/2378677303 >> v2:172.21.15.191:3300/0 conn(0x7ff4bc0c7fc0 0x7ff4bc0c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f6450c81640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f64440d9240 0x7f6444039e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fd926ffd640 1 -- 172.21.15.186:0/2993624802 >> v2:172.21.15.191:3300/0 conn(0x7fd9281567b0 msgr2=0x7fd9281505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fd926ffd640 1 --2- 172.21.15.186:0/2993624802 >> v2:172.21.15.191:3300/0 conn(0x7fd9281567b0 0x7fd9281505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fd926ffd640 1 -- 172.21.15.186:0/2993624802 >> v1:172.21.15.186:6789/0 conn(0x7fd92806f230 legacy=0x7fd92806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fd926ffd640 1 -- 172.21.15.186:0/2993624802 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd928156ff0 con 0x7fd92806ee50 2024-03-28T15:01:37.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7ff4c1c21640 1 --2- 172.21.15.186:0/2378677303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4bc16b620 0x7ff4bc06cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7ff4c1c21640 1 --2- 172.21.15.186:0/2378677303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4bc073220 0x7ff4bc0c6b60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.619+0000 7ff4c1c21640 1 -- 172.21.15.186:0/2378677303 >> 172.21.15.186:0/2378677303 conn(0x7ff4bc06b460 msgr2=0x7ff4bc06b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f6450c81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f644404ffc0 0x7f64440dfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f644a59c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f64440d9240 0x7f6444039e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fabc359e640 1 -- 172.21.15.186:0/1838894111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabbc04c690 msgr2=0x7fabbc0dd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fabc359e640 1 --2- 172.21.15.186:0/1838894111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabbc04c690 0x7fabbc0dd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fabc359e640 1 -- 172.21.15.186:0/1838894111 >> v2:172.21.15.191:3300/0 conn(0x7fabbc0d5e50 msgr2=0x7fabbc0dcf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fabc359e640 1 --2- 172.21.15.186:0/1838894111 >> v2:172.21.15.191:3300/0 conn(0x7fabbc0d5e50 0x7fabbc0dcf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fabc359e640 1 -- 172.21.15.186:0/1838894111 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabbc008730 con 0x7fabbc0defb0 2024-03-28T15:01:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f644a59c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f64440d9240 0x7f6444039e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58248/0 (socket says 172.21.15.186:58248) 2024-03-28T15:01:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f644a59c640 1 -- 172.21.15.186:0/4240060058 learned_addr learned my addr 172.21.15.186:0/4240060058 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f6450c81640 1 -- 172.21.15.186:0/4240060058 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f644403a4a0 con 0x7f64440e0480 2024-03-28T15:01:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f6450c81640 1 -- 172.21.15.186:0/4240060058 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f64440dffd0 con 0x7f64440d9240 2024-03-28T15:01:37.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f6450c81640 1 -- 172.21.15.186:0/4240060058 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f6444050450 con 0x7f644404ffc0 2024-03-28T15:01:37.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fd7bed9d640 1 -- 172.21.15.186:0/1912442024 >> v2:172.21.15.191:3300/0 conn(0x7fd7c00c8a20 msgr2=0x7fd7c006ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fd7bed9d640 1 --2- 172.21.15.186:0/1912442024 >> v2:172.21.15.191:3300/0 conn(0x7fd7c00c8a20 0x7fd7c006ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fd7bed9d640 1 -- 172.21.15.186:0/1912442024 >> v1:172.21.15.191:6789/0 conn(0x7fd7c00cb6d0 legacy=0x7fd7c0071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7fd7bed9d640 1 -- 172.21.15.186:0/1912442024 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c00c4f90 con 0x7fd7c0152960 2024-03-28T15:01:37.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f8bbba15640 1 -- start start 2024-03-28T15:01:37.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f2102ffd640 1 -- 172.21.15.186:0/2495608779 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f8034070 con 0x7f20fc0452c0 2024-03-28T15:01:37.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f2102ffd640 1 -- 172.21.15.186:0/2495608779 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20f8030850 con 0x7f20fc0452c0 2024-03-28T15:01:37.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f890f238640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f89000dd760 0x7f890004e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f890f238640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f890004cd10 0x7f89000e3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f890f238640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f890004e860 con 0x7f8900007e40 2024-03-28T15:01:37.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f890f238640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f890004eb40 con 0x7f89000dd760 2024-03-28T15:01:37.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f890f238640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8900008670 con 0x7f890004cd10 2024-03-28T15:01:37.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f890e236640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f89000dd760 0x7f890004e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f890e236640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f89000dd760 0x7f890004e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58256/0 (socket says 172.21.15.186:58256) 2024-03-28T15:01:37.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f890e236640 1 -- 172.21.15.186:0/2619042702 learned_addr learned my addr 172.21.15.186:0/2619042702 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f890ea37640 1 --2- 172.21.15.186:0/2619042702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f890004cd10 0x7f89000e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7fabc359e640 1 --2- 172.21.15.186:0/1838894111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc0defb0 0x7fabbc0df400 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fabbc04ebd0 tx=0x7fabac02fc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5116c361169076d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7fd926ffd640 1 --2- 172.21.15.186:0/2993624802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd92806ee50 0x7fd9280c8370 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd920009120 tx=0x7fd92002bfe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbeafba6d0b6a558 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f8bbba15640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bb4072770 0x7f8bb40cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f8bbba15640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8bb415ad70 0x7f8bb415b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f8bbba15640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bb415e0e0 con 0x7f8bb4154910 2024-03-28T15:01:37.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f8bbba15640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8bb415b720 con 0x7f8bb415ad70 2024-03-28T15:01:37.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f8bbba15640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8bb40cc7a0 con 0x7f8bb4072770 2024-03-28T15:01:37.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7fd7bed9d640 1 --2- 172.21.15.186:0/1912442024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c0152960 0x7fd7c006e810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00ceec0 tx=0x7fd7b002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1ea9c871c271cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f644a59c640 1 -- 172.21.15.186:0/4240060058 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f644404ffc0 msgr2=0x7f64440dfa90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T15:01:37.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f644a59c640 1 --2- 172.21.15.186:0/4240060058 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f644404ffc0 0x7f64440dfa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f644a59c640 1 -- 172.21.15.186:0/4240060058 >> v1:172.21.15.191:6789/0 conn(0x7f64440e0480 legacy=0x7f64440071a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.620+0000 7f644a59c640 1 -- 172.21.15.186:0/4240060058 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f644404ef20 con 0x7f64440d9240 2024-03-28T15:01:37.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f2102ffd640 1 -- 172.21.15.186:0/2495608779 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f8030c20 con 0x7f20fc0452c0 2024-03-28T15:01:37.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f210a66f640 1 -- 172.21.15.186:0/2495608779 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20fc0452c0 msgr2=0x7f20fc0456e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f210a66f640 1 --2- 172.21.15.186:0/2495608779 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20fc0452c0 0x7f20fc0456e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f20fc008520 tx=0x7f20f802b990 comp rx=0 tx=0).stop 2024-03-28T15:01:37.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f8bb97b1640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8bb4154910 0x7f8bb40d0460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:37448/0 (socket says 172.21.15.186:37448) 2024-03-28T15:01:37.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f8bb97b1640 1 -- 172.21.15.186:0/3164806155 learned_addr learned my addr 172.21.15.186:0/3164806155 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:37.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f8bb9fb2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8bb415ad70 0x7f8bb415b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7fd7bdd9b640 1 -- 172.21.15.186:0/1912442024 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0034070 con 0x7fd7c0152960 2024-03-28T15:01:37.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7fd7bdd9b640 1 -- 172.21.15.186:0/1912442024 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b0030610 con 0x7fd7c0152960 2024-03-28T15:01:37.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7fd7bdd9b640 1 -- 172.21.15.186:0/1912442024 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b00309e0 con 0x7fd7c0152960 2024-03-28T15:01:37.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7fabc1d9b640 1 -- 172.21.15.186:0/1838894111 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabac03d070 con 0x7fabbc0defb0 2024-03-28T15:01:37.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7fabc1d9b640 1 -- 172.21.15.186:0/1838894111 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabac030a00 con 0x7fabbc0defb0 2024-03-28T15:01:37.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7fabc1d9b640 1 -- 172.21.15.186:0/1838894111 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabac030dd0 con 0x7fabbc0defb0 2024-03-28T15:01:37.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f8bb8fb0640 1 --2- 172.21.15.186:0/3164806155 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bb4072770 0x7f8bb40cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f644a59c640 1 --2- 172.21.15.186:0/4240060058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440d9240 0x7f6444039e60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f643c009870 tx=0x7f643c02ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d18dfc413b23d6e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7f890e236640 1 -- 172.21.15.186:0/2619042702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f890004cd10 msgr2=0x7f89000e3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f890e236640 1 --2- 172.21.15.186:0/2619042702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f890004cd10 0x7f89000e3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f890e236640 1 -- 172.21.15.186:0/2619042702 >> v1:172.21.15.186:6789/0 conn(0x7f8900007e40 legacy=0x7f890004d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f890e236640 1 -- 172.21.15.186:0/2619042702 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8900008950 con 0x7f89000dd760 2024-03-28T15:01:37.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7ff4c1c21640 1 -- 172.21.15.186:0/2378677303 shutdown_connections 2024-03-28T15:01:37.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.621+0000 7fd925ffb640 1 -- 172.21.15.186:0/2993624802 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92000c040 con 0x7fd92806ee50 2024-03-28T15:01:37.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f210a66f640 1 -- 172.21.15.186:0/2495608779 shutdown_connections 2024-03-28T15:01:37.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f210a66f640 1 --2- 172.21.15.186:0/2495608779 >> v2:172.21.15.186:3300/0 conn(0x7f20fc0e6760 0x7f20fc0e6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f210a66f640 1 --2- 172.21.15.186:0/2495608779 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20fc0452c0 0x7f20fc0456e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f210a66f640 1 --2- 172.21.15.186:0/2495608779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20fc0dd440 0x7f20fc04b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f210a66f640 1 -- 172.21.15.186:0/2495608779 >> 172.21.15.186:0/2495608779 conn(0x7f20fc050170 msgr2=0x7f20fc050580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7fd925ffb640 1 -- 172.21.15.186:0/2993624802 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd920003b90 con 0x7fd92806ee50 2024-03-28T15:01:37.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f8bb8fb0640 1 -- 172.21.15.186:0/3164806155 >> v2:172.21.15.186:3300/0 conn(0x7f8bb415ad70 msgr2=0x7f8bb415b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f8bb8fb0640 1 --2- 172.21.15.186:0/3164806155 >> v2:172.21.15.186:3300/0 conn(0x7f8bb415ad70 0x7f8bb415b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f8bb8fb0640 1 -- 172.21.15.186:0/3164806155 >> v1:172.21.15.186:6789/0 conn(0x7f8bb4154910 legacy=0x7f8bb40d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7fabc90f4640 1 -- 172.21.15.186:0/1838894111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc0defb0 msgr2=0x7fabbc0df400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7fabc90f4640 1 --2- 172.21.15.186:0/1838894111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc0defb0 0x7fabbc0df400 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fabbc04ebd0 tx=0x7fabac02fc50 comp rx=0 tx=0).stop 2024-03-28T15:01:37.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7ff4c1c21640 1 -- 172.21.15.186:0/2378677303 wait complete. 2024-03-28T15:01:37.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f210a66f640 1 -- 172.21.15.186:0/2495608779 shutdown_connections 2024-03-28T15:01:37.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7fabc90f4640 1 -- 172.21.15.186:0/1838894111 shutdown_connections 2024-03-28T15:01:37.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7fabc90f4640 1 --2- 172.21.15.186:0/1838894111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc0defb0 0x7fabbc0df400 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7fabc90f4640 1 --2- 172.21.15.186:0/1838894111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabbc04c690 0x7fabbc0dd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7fabc90f4640 1 --2- 172.21.15.186:0/1838894111 >> v2:172.21.15.191:3300/0 conn(0x7fabbc0d5e50 0x7fabbc0dcf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7fabc90f4640 1 -- 172.21.15.186:0/1838894111 >> 172.21.15.186:0/1838894111 conn(0x7fabbc04dd60 msgr2=0x7fabbc04e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7fd925ffb640 1 -- 172.21.15.186:0/2993624802 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92002c980 con 0x7fd92806ee50 2024-03-28T15:01:37.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd92d9f7640 1 -- 172.21.15.186:0/2993624802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd92806ee50 msgr2=0x7fd9280c8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd92d9f7640 1 --2- 172.21.15.186:0/2993624802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd92806ee50 0x7fd9280c8370 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd920009120 tx=0x7fd92002bfe0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7f890e236640 1 --2- 172.21.15.186:0/2619042702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000dd760 0x7f890004e1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f89040098a0 tx=0x7f890402b720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da73aa71d5ab0b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fabc90f4640 1 -- 172.21.15.186:0/1838894111 shutdown_connections 2024-03-28T15:01:37.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fabc90f4640 1 -- 172.21.15.186:0/1838894111 wait complete. 2024-03-28T15:01:37.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7fd7c504e640 1 -- 172.21.15.186:0/1912442024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c0152960 msgr2=0x7fd7c006e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7fd7c504e640 1 --2- 172.21.15.186:0/1912442024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c0152960 0x7fd7c006e810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00ceec0 tx=0x7fd7b002b960 comp rx=0 tx=0).stop 2024-03-28T15:01:37.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd7c504e640 1 -- 172.21.15.186:0/1912442024 shutdown_connections 2024-03-28T15:01:37.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd7c504e640 1 --2- 172.21.15.186:0/1912442024 >> v2:172.21.15.191:3300/0 conn(0x7fd7c00c8a20 0x7fd7c006ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd7c504e640 1 --2- 172.21.15.186:0/1912442024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c0152960 0x7fd7c006e810 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00ceec0 tx=0x7fd7b002b960 comp rx=0 tx=0).stop 2024-03-28T15:01:37.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd7c504e640 1 -- 172.21.15.186:0/1912442024 >> 172.21.15.186:0/1912442024 conn(0x7fd7c006fa00 msgr2=0x7fd7c0073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7f644959a640 1 -- 172.21.15.186:0/4240060058 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c02f850 con 0x7f64440d9240 2024-03-28T15:01:37.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7f644959a640 1 -- 172.21.15.186:0/4240060058 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f643c02f9f0 con 0x7f64440d9240 2024-03-28T15:01:37.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd92d9f7640 1 -- 172.21.15.186:0/2993624802 shutdown_connections 2024-03-28T15:01:37.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd92d9f7640 1 --2- 172.21.15.186:0/2993624802 >> v2:172.21.15.191:3300/0 conn(0x7fd9281567b0 0x7fd9281505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd92d9f7640 1 --2- 172.21.15.186:0/2993624802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd92806ee50 0x7fd9280c8370 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd920009120 tx=0x7fd92002bfe0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd92d9f7640 1 -- 172.21.15.186:0/2993624802 >> 172.21.15.186:0/2993624802 conn(0x7fd92806c010 msgr2=0x7fd9280cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7f644959a640 1 -- 172.21.15.186:0/4240060058 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c02fdc0 con 0x7f64440d9240 2024-03-28T15:01:37.649 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7f890d234640 1 -- 172.21.15.186:0/2619042702 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904037070 con 0x7f89000dd760 2024-03-28T15:01:37.649 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7f890d234640 1 -- 172.21.15.186:0/2619042702 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8904030710 con 0x7f89000dd760 2024-03-28T15:01:37.649 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.622+0000 7f8bb8fb0640 1 -- 172.21.15.186:0/3164806155 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb415e3a0 con 0x7f8bb4072770 2024-03-28T15:01:37.649 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd7c504e640 1 -- 172.21.15.186:0/1912442024 shutdown_connections 2024-03-28T15:01:37.649 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7f210a66f640 1 -- 172.21.15.186:0/2495608779 wait complete. 2024-03-28T15:01:37.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd92d9f7640 1 -- 172.21.15.186:0/2993624802 shutdown_connections 2024-03-28T15:01:37.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7fd92d9f7640 1 -- 172.21.15.186:0/2993624802 wait complete. 2024-03-28T15:01:37.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7f8bb8fb0640 1 --2- 172.21.15.186:0/3164806155 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bb4072770 0x7f8bb40cc260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40097a0 tx=0x7f8ba402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=422a852ae2708840 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f6450c81640 1 -- 172.21.15.186:0/4240060058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440d9240 msgr2=0x7f6444039e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f6450c81640 1 --2- 172.21.15.186:0/4240060058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440d9240 0x7f6444039e60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f643c009870 tx=0x7f643c02ba30 comp rx=0 tx=0).stop 2024-03-28T15:01:37.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f890d234640 1 -- 172.21.15.186:0/2619042702 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904030ae0 con 0x7f89000dd760 2024-03-28T15:01:37.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.623+0000 7ff4c1c21640 1 Processor -- start 2024-03-28T15:01:37.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7ff4c1c21640 1 -- start start 2024-03-28T15:01:37.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f8babfff640 1 -- 172.21.15.186:0/3164806155 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4034070 con 0x7f8bb4072770 2024-03-28T15:01:37.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f8babfff640 1 -- 172.21.15.186:0/3164806155 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba4030800 con 0x7f8bb4072770 2024-03-28T15:01:37.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f8babfff640 1 -- 172.21.15.186:0/3164806155 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4030bd0 con 0x7f8bb4072770 2024-03-28T15:01:37.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7fd7c504e640 1 -- 172.21.15.186:0/1912442024 wait complete. 2024-03-28T15:01:37.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f890f238640 1 -- 172.21.15.186:0/2619042702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000dd760 msgr2=0x7f890004e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f890f238640 1 --2- 172.21.15.186:0/2619042702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000dd760 0x7f890004e1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f89040098a0 tx=0x7f890402b720 comp rx=0 tx=0).stop 2024-03-28T15:01:37.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f890f238640 1 -- 172.21.15.186:0/2619042702 shutdown_connections 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f890f238640 1 --2- 172.21.15.186:0/2619042702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f890004cd10 0x7f89000e3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f890f238640 1 --2- 172.21.15.186:0/2619042702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000dd760 0x7f890004e1b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f890f238640 1 -- 172.21.15.186:0/2619042702 >> 172.21.15.186:0/2619042702 conn(0x7f890000e690 msgr2=0x7f8900009d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f890f238640 1 -- 172.21.15.186:0/2619042702 shutdown_connections 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7fabc90f4640 1 Processor -- start 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7fabc90f4640 1 -- start start 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fabc90f4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fabbc04c690 0x7fabbc171000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fabc90f4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc0d5e50 0x7fabbc0d2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fabc2d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fabbc04c690 0x7fabbc171000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fabc259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc0d5e50 0x7fabbc0d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fabc2d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fabbc04c690 0x7fabbc171000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58286/0 (socket says 172.21.15.186:58286) 2024-03-28T15:01:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fabc2d9d640 1 -- 172.21.15.186:0/4108234262 learned_addr learned my addr 172.21.15.186:0/4108234262 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fabc90f4640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabbc0defb0 0x7fabbc1742a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fabbc008f90 con 0x7fabbc0d5e50 2024-03-28T15:01:37.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fabbc008cb0 con 0x7fabbc04c690 2024-03-28T15:01:37.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fabbc04d860 con 0x7fabbc0defb0 2024-03-28T15:01:37.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7fd92d9f7640 1 Processor -- start 2024-03-28T15:01:37.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7fd92d9f7640 1 -- start start 2024-03-28T15:01:37.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fd92d9f7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd92806f230 0x7fd928148e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7fd92d9f7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9281567b0 0x7fd928142c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7fd92d9f7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd928147d40 0x7fd9281f8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f6450c81640 1 -- 172.21.15.186:0/4240060058 shutdown_connections 2024-03-28T15:01:37.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f6450c81640 1 --2- 172.21.15.186:0/4240060058 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f644404ffc0 0x7f64440dfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f6450c81640 1 --2- 172.21.15.186:0/4240060058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440d9240 0x7f6444039e60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f6450c81640 1 -- 172.21.15.186:0/4240060058 >> 172.21.15.186:0/4240060058 conn(0x7f644400e150 msgr2=0x7f644400e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7ff4c1c21640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4bc073220 0x7ff4bc200e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7ff4c1c21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4bc0c7fc0 0x7ff4bc164230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7ff4c1c21640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4bc16b620 0x7ff4bc162320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7ff4c1c21640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc06d320 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7ff4c1c21640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc16ae20 con 0x7ff4bc16b620 2024-03-28T15:01:37.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7ff4c1c21640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff4bc06e460 con 0x7ff4bc073220 2024-03-28T15:01:37.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7ff4bbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4bc0c7fc0 0x7ff4bc164230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7ff4bbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4bc0c7fc0 0x7ff4bc164230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41300/0 (socket says 172.21.15.186:41300) 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7ff4bbfff640 1 -- 172.21.15.186:0/1470959126 learned_addr learned my addr 172.21.15.186:0/1470959126 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7ff4c1420640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4bc16b620 0x7ff4bc162320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7ff4c0c1f640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4bc073220 0x7ff4bc200e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4bbfff640 1 -- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4bc073220 msgr2=0x7ff4bc200e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4bbfff640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4bc073220 0x7ff4bc200e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4bbfff640 1 -- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4bc16b620 msgr2=0x7ff4bc162320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4bbfff640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4bc16b620 0x7ff4bc162320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4bbfff640 1 -- 172.21.15.186:0/1470959126 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4bc201560 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4c1420640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4bc16b620 0x7ff4bc162320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4bbfff640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4bc0c7fc0 0x7ff4bc164230 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac00b3e0 tx=0x7ff4ac00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4b97fa640 1 -- 172.21.15.186:0/1470959126 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac015020 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4b97fa640 1 -- 172.21.15.186:0/1470959126 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4ac011850 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4b97fa640 1 -- 172.21.15.186:0/1470959126 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac011c40 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4c1c21640 1 -- 172.21.15.186:0/1470959126 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4bc201840 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f210a66f640 1 Processor -- start 2024-03-28T15:01:37.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f210a66f640 1 -- start start 2024-03-28T15:01:37.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f210a66f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20fc0452c0 0x7f20fc0e3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f210a66f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20fc0dd440 0x7f20fc0e5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f210a66f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20fc0e6760 0x7f20fc0d53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f210a66f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc045c20 con 0x7f20fc0452c0 2024-03-28T15:01:37.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f210a66f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc0072d0 con 0x7f20fc0dd440 2024-03-28T15:01:37.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f210a66f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f20fc0dd820 con 0x7f20fc0e6760 2024-03-28T15:01:37.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f8bbba15640 1 -- 172.21.15.186:0/3164806155 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bb4072770 msgr2=0x7f8bb40cc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f8bbba15640 1 --2- 172.21.15.186:0/3164806155 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bb4072770 0x7f8bb40cc260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40097a0 tx=0x7f8ba402b940 comp rx=0 tx=0).stop 2024-03-28T15:01:37.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f8bbba15640 1 -- 172.21.15.186:0/3164806155 shutdown_connections 2024-03-28T15:01:37.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f8bbba15640 1 --2- 172.21.15.186:0/3164806155 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bb4072770 0x7f8bb40cc260 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40097a0 tx=0x7f8ba402b940 comp rx=0 tx=0).stop 2024-03-28T15:01:37.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f8bbba15640 1 --2- 172.21.15.186:0/3164806155 >> v2:172.21.15.186:3300/0 conn(0x7f8bb415ad70 0x7f8bb415b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.625+0000 7f8bbba15640 1 -- 172.21.15.186:0/3164806155 >> 172.21.15.186:0/3164806155 conn(0x7f8bb406b880 msgr2=0x7f8bb406bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f8bbba15640 1 -- 172.21.15.186:0/3164806155 shutdown_connections 2024-03-28T15:01:37.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fd7c504e640 1 Processor -- start 2024-03-28T15:01:37.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7fd7c504e640 1 -- start start 2024-03-28T15:01:37.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7fabc359e640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabbc0defb0 0x7fabbc1742a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7fd92d9f7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd92806d7c0 con 0x7fd92806f230 2024-03-28T15:01:37.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7fd92d9f7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd9281575b0 con 0x7fd928147d40 2024-03-28T15:01:37.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7fd92d9f7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd92806d940 con 0x7fd9281567b0 2024-03-28T15:01:37.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7ff4c1c21640 1 -- 172.21.15.186:0/1470959126 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4bc161b20 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7fd926ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd92806f230 0x7fd928148e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7fd926ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd92806f230 0x7fd928148e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41330/0 (socket says 172.21.15.186:41330) 2024-03-28T15:01:37.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7fd926ffd640 1 -- 172.21.15.186:0/4201696495 learned_addr learned my addr 172.21.15.186:0/4201696495 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f8bbba15640 1 -- 172.21.15.186:0/3164806155 wait complete. 2024-03-28T15:01:37.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f21037fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20fc0dd440 0x7f20fc0e5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f21037fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20fc0dd440 0x7f20fc0e5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58292/0 (socket says 172.21.15.186:58292) 2024-03-28T15:01:37.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f21037fe640 1 -- 172.21.15.186:0/942415693 learned_addr learned my addr 172.21.15.186:0/942415693 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7fd7c504e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c00c8a20 0x7fd7c015a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7fd7c504e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c00cb6d0 0x7fd7c0151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7f21037fe640 1 -- 172.21.15.186:0/942415693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20fc0e6760 msgr2=0x7f20fc0d53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7c504e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c01541b0 0x7fd7c01f0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd926ffd640 1 -- 172.21.15.186:0/4201696495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9281567b0 msgr2=0x7fd928142c70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T15:01:37.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd926ffd640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9281567b0 0x7fd928142c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd926ffd640 1 -- 172.21.15.186:0/4201696495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd928147d40 msgr2=0x7fd9281f8890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T15:01:37.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd926ffd640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd928147d40 0x7fd9281f8890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd926ffd640 1 -- 172.21.15.186:0/4201696495 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9280c5f20 con 0x7fd92806f230 2024-03-28T15:01:37.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7be59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c00cb6d0 0x7fd7c0151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7be59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c00cb6d0 0x7fd7c0151440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58304/0 (socket says 172.21.15.186:58304) 2024-03-28T15:01:37.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7be59c640 1 -- 172.21.15.186:0/3128938452 learned_addr learned my addr 172.21.15.186:0/3128938452 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c00c31c0 con 0x7fd7c00c8a20 2024-03-28T15:01:37.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.624+0000 7f890f238640 1 -- 172.21.15.186:0/2619042702 wait complete. 2024-03-28T15:01:37.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c00c3040 con 0x7fd7c00cb6d0 2024-03-28T15:01:37.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c00c8390 con 0x7fd7c01541b0 2024-03-28T15:01:37.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f890f238640 1 Processor -- start 2024-03-28T15:01:37.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.626+0000 7f890f238640 1 -- start start 2024-03-28T15:01:37.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7bf59e640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c01541b0 0x7fd7c01f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7bed9d640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c00c8a20 0x7fd7c015a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd926ffd640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd92806f230 0x7fd928148e80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd920000f80 tx=0x7fd920004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7f21037fe640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20fc0e6760 0x7f20fc0d53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7f21037fe640 1 -- 172.21.15.186:0/942415693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20fc0452c0 msgr2=0x7f20fc0e3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f6450c81640 1 -- 172.21.15.186:0/4240060058 shutdown_connections 2024-03-28T15:01:37.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7f21037fe640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20fc0452c0 0x7f20fc0e3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7f21037fe640 1 -- 172.21.15.186:0/942415693 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20fc178fc0 con 0x7f20fc0dd440 2024-03-28T15:01:37.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7ff4c1c21640 1 -- 172.21.15.186:0/1470959126 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff4bc201d60 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f6450c81640 1 -- 172.21.15.186:0/4240060058 wait complete. 2024-03-28T15:01:37.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f21037fe640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20fc0dd440 0x7f20fc0e5e40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f20f8000c00 tx=0x7f20f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7be59c640 1 -- 172.21.15.186:0/3128938452 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c01541b0 msgr2=0x7fd7c01f0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7be59c640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c01541b0 0x7fd7c01f0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7be59c640 1 -- 172.21.15.186:0/3128938452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c00c8a20 msgr2=0x7fd7c015a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7be59c640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c00c8a20 0x7fd7c015a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd7be59c640 1 -- 172.21.15.186:0/3128938452 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c01f1280 con 0x7fd7c00cb6d0 2024-03-28T15:01:37.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fd91ffff640 1 -- 172.21.15.186:0/4201696495 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd920043070 con 0x7fd92806f230 2024-03-28T15:01:37.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fabc2d9d640 1 -- 172.21.15.186:0/4108234262 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabbc0defb0 msgr2=0x7fabbc1742a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fabc2d9d640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabbc0defb0 0x7fabbc1742a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fabc2d9d640 1 -- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc0d5e50 msgr2=0x7fabbc0d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fabc2d9d640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc0d5e50 0x7fabbc0d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7fabc2d9d640 1 -- 172.21.15.186:0/4108234262 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabbc171540 con 0x7fabbc04c690 2024-03-28T15:01:37.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f2100ff9640 1 -- 172.21.15.186:0/942415693 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f8034040 con 0x7f20fc0dd440 2024-03-28T15:01:37.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7fd92d9f7640 1 -- 172.21.15.186:0/4201696495 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9281f5960 con 0x7fd92806f230 2024-03-28T15:01:37.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f210a66f640 1 -- 172.21.15.186:0/942415693 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20fc179280 con 0x7f20fc0dd440 2024-03-28T15:01:37.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7fd92d9f7640 1 -- 172.21.15.186:0/4201696495 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd928148570 con 0x7fd92806f230 2024-03-28T15:01:37.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7f8bbba15640 1 Processor -- start 2024-03-28T15:01:37.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.627+0000 7f8bbba15640 1 -- start start 2024-03-28T15:01:37.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bbba15640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bb4154910 0x7f8bb4151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bbba15640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bb415ad70 0x7f8bb40ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bbba15640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bb415b900 0x7f8bb41f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bbba15640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb415e960 con 0x7f8bb415ad70 2024-03-28T15:01:37.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bbba15640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb415a8e0 con 0x7f8bb415b900 2024-03-28T15:01:37.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bbba15640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8bb415ec40 con 0x7f8bb4154910 2024-03-28T15:01:37.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7fd7be59c640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c00cb6d0 0x7fd7c0151440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c3f0 tx=0x7fd7b400c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890f238640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8900007e40 0x7f89000361e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7fd91ffff640 1 -- 172.21.15.186:0/4201696495 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd92000c000 con 0x7fd92806f230 2024-03-28T15:01:37.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7fd91ffff640 1 -- 172.21.15.186:0/4201696495 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd920004670 con 0x7fd92806f230 2024-03-28T15:01:37.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7fd92d9f7640 1 -- 172.21.15.186:0/4201696495 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd90c001030 con 0x7fd92806f230 2024-03-28T15:01:37.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7f6450c81640 1 Processor -- start 2024-03-28T15:01:37.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7f6450c81640 1 -- start start 2024-03-28T15:01:37.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890f238640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f890004cd10 0x7f89000da780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890f238640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000e2e00 0x7f8900052370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890f238640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f890004df00 con 0x7f8900007e40 2024-03-28T15:01:37.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890f238640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f890004dd80 con 0x7f890004cd10 2024-03-28T15:01:37.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890f238640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f89000e3a70 con 0x7f89000e2e00 2024-03-28T15:01:37.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890ea37640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000e2e00 0x7f8900052370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890ea37640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000e2e00 0x7f8900052370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:56976/0 (socket says 172.21.15.186:56976) 2024-03-28T15:01:37.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890ea37640 1 -- 172.21.15.186:0/2525547612 learned_addr learned my addr 172.21.15.186:0/2525547612 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890e236640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8900007e40 0x7f89000361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890ea37640 1 -- 172.21.15.186:0/2525547612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f890004cd10 msgr2=0x7f89000da780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T15:01:37.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890ea37640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f890004cd10 0x7f89000da780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.628+0000 7f890ea37640 1 -- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8900007e40 msgr2=0x7f89000361e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f890ea37640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8900007e40 0x7f89000361e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f890ea37640 1 -- 172.21.15.186:0/2525547612 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8900035390 con 0x7f89000e2e00 2024-03-28T15:01:37.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f890e236640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8900007e40 0x7f89000361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:37.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f890ea37640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000e2e00 0x7f8900052370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8908050b20 tx=0x7f89080691b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7f88feffd640 1 -- 172.21.15.186:0/2525547612 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908089070 con 0x7f89000e2e00 2024-03-28T15:01:37.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7f88feffd640 1 -- 172.21.15.186:0/2525547612 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f890806cb20 con 0x7f89000e2e00 2024-03-28T15:01:37.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7ff4b97fa640 1 -- 172.21.15.186:0/1470959126 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff4ac018020 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7ff4b97fa640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff4a806c530 0x7ff4a806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7ff4b97fa640 1 -- 172.21.15.186:0/1470959126 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff4ac08b5d0 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7ff4b97fa640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] conn(0x7ff4a8074730 0x7ff4a8076b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7ff4b97fa640 1 -- 172.21.15.186:0/1470959126 --> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4ac020ec0 con 0x7ff4a8074730 2024-03-28T15:01:37.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7ff4b97fa640 1 -- 172.21.15.186:0/1470959126 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff4ac092050 con 0x7ff4bc0c7fc0 2024-03-28T15:01:37.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7ff4c0c1f640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff4a806c530 0x7ff4a806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7ff4c1420640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] conn(0x7ff4a8074730 0x7ff4a8076b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f210a66f640 1 -- 172.21.15.186:0/942415693 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20fc179840 con 0x7f20fc0dd440 2024-03-28T15:01:37.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f2100ff9640 1 -- 172.21.15.186:0/942415693 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20f8039c90 con 0x7f20fc0dd440 2024-03-28T15:01:37.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7f2100ff9640 1 -- 172.21.15.186:0/942415693 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f8043630 con 0x7f20fc0dd440 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bb8fb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bb415ad70 0x7f8bb40ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bb8fb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bb415ad70 0x7f8bb40ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41370/0 (socket says 172.21.15.186:41370) 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bb8fb0640 1 -- 172.21.15.186:0/535902944 learned_addr learned my addr 172.21.15.186:0/535902944 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bb97b1640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bb4154910 0x7f8bb4151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bb8fb0640 1 -- 172.21.15.186:0/535902944 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bb4154910 msgr2=0x7f8bb4151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bb8fb0640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bb4154910 0x7f8bb4151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bb8fb0640 1 -- 172.21.15.186:0/535902944 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bb415b900 msgr2=0x7f8bb41f3b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bb8fb0640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bb415b900 0x7f8bb41f3b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7f8bb8fb0640 1 -- 172.21.15.186:0/535902944 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb415d5c0 con 0x7f8bb415ad70 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7f8bb8fb0640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bb415ad70 0x7f8bb40ccb90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40098c0 tx=0x7f8ba40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7f8ba9ffb640 1 -- 172.21.15.186:0/535902944 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4034040 con 0x7f8bb415ad70 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7f8ba9ffb640 1 -- 172.21.15.186:0/535902944 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba4039b80 con 0x7f8bb415ad70 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7f8ba9ffb640 1 -- 172.21.15.186:0/535902944 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40448d0 con 0x7f8bb415ad70 2024-03-28T15:01:37.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.630+0000 7f8bbba15640 1 -- 172.21.15.186:0/535902944 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb4152300 con 0x7f8bb415ad70 2024-03-28T15:01:37.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7fd7a77fe640 1 -- 172.21.15.186:0/3128938452 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4017070 con 0x7fd7c00cb6d0 2024-03-28T15:01:37.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7fd7a77fe640 1 -- 172.21.15.186:0/3128938452 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b4005ce0 con 0x7fd7c00cb6d0 2024-03-28T15:01:37.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7fd7a77fe640 1 -- 172.21.15.186:0/3128938452 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b40072f0 con 0x7fd7c00cb6d0 2024-03-28T15:01:37.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.629+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c01f1560 con 0x7fd7c00cb6d0 2024-03-28T15:01:37.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7f890f238640 1 -- 172.21.15.186:0/2525547612 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f890019e490 con 0x7f89000e2e00 2024-03-28T15:01:37.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7f890f238640 1 -- 172.21.15.186:0/2525547612 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f890019ea50 con 0x7f89000e2e00 2024-03-28T15:01:37.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7f88feffd640 1 -- 172.21.15.186:0/2525547612 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908084430 con 0x7f89000e2e00 2024-03-28T15:01:37.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7fd7a77fe640 1 -- 172.21.15.186:0/3128938452 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd7b401a070 con 0x7fd7c00cb6d0 2024-03-28T15:01:37.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7fabc2d9d640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fabbc04c690 0x7fabbc171000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fabc4069de0 tx=0x7fabc40691b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7fabbb7fe640 1 -- 172.21.15.186:0/4108234262 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc406cba0 con 0x7fabbc04c690 2024-03-28T15:01:37.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabbc0cccd0 con 0x7fabbc04c690 2024-03-28T15:01:37.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabbc0cd270 con 0x7fabbc04c690 2024-03-28T15:01:37.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7fabbb7fe640 1 -- 172.21.15.186:0/4108234262 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabc409b030 con 0x7fabbc04c690 2024-03-28T15:01:37.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7fabbb7fe640 1 -- 172.21.15.186:0/4108234262 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4097890 con 0x7fabbc04c690 2024-03-28T15:01:37.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7f20ee7fc640 1 -- 172.21.15.186:0/942415693 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f20fc008120 con 0x7f20fc0dd440 2024-03-28T15:01:37.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.632+0000 7f6450c81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f644404ffc0 0x7f6444169380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.632+0000 7ff4c0c1f640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff4a806c530 0x7ff4a806e9f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff4b403b000 tx=0x7ff4b4036000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.632+0000 7f2100ff9640 1 -- 172.21.15.186:0/942415693 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f20f804a060 con 0x7f20fc0dd440 2024-03-28T15:01:37.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.632+0000 7fd91ffff640 1 -- 172.21.15.186:0/4201696495 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd920046070 con 0x7fd92806f230 2024-03-28T15:01:37.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.632+0000 7fd7a77fe640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7ac06c580 0x7fd7ac06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.632+0000 7ff4c1420640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] conn(0x7ff4a8074730 0x7ff4a8076b50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.632+0000 7f890f238640 1 -- 172.21.15.186:0/2525547612 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f89000057d0 con 0x7f89000e2e00 2024-03-28T15:01:37.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.632+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c01598b0 con 0x7fd7c00cb6d0 2024-03-28T15:01:37.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.632+0000 7fd7bed9d640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7ac06c580 0x7fd7ac06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f2100ff9640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f20e806c630 0x7f20e806eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.632+0000 7f6450c81640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64440d9240 0x7f6444169f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f6450c81640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440e0480 0x7f644416a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f6450c81640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f644404f7c0 con 0x7f64440d9240 2024-03-28T15:01:37.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f6450c81640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f644404f4e0 con 0x7f64440e0480 2024-03-28T15:01:37.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f6450c81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f644404faa0 con 0x7f644404ffc0 2024-03-28T15:01:37.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f2103fff640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f20e806c630 0x7f20e806eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f2100ff9640 1 -- 172.21.15.186:0/942415693 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f20f80ad5d0 con 0x7f20fc0dd440 2024-03-28T15:01:37.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f644ad9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440e0480 0x7f644416a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f644ad9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440e0480 0x7f644416a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58318/0 (socket says 172.21.15.186:58318) 2024-03-28T15:01:37.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f644ad9d640 1 -- 172.21.15.186:0/3936275979 learned_addr learned my addr 172.21.15.186:0/3936275979 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f2100ff9640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] conn(0x7f20e80748b0 0x7f20e8076cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f2100ff9640 1 -- 172.21.15.186:0/942415693 --> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20f8001d30 con 0x7f20e80748b0 2024-03-28T15:01:37.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f2100ff9640 1 -- 172.21.15.186:0/942415693 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f20f80b4050 con 0x7f20fc0dd440 2024-03-28T15:01:37.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7fabbb7fe640 1 -- 172.21.15.186:0/4108234262 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fabc4097a30 con 0x7fabbc04c690 2024-03-28T15:01:37.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7fd7a77fe640 1 -- 172.21.15.186:0/3128938452 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd7b4056750 con 0x7fd7c00cb6d0 2024-03-28T15:01:37.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f6449d9b640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64440d9240 0x7f6444169f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f644a59c640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f644404ffc0 0x7f6444169380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f6449d9b640 1 -- 172.21.15.186:0/3936275979 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f644404ffc0 msgr2=0x7f6444169380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f6449d9b640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f644404ffc0 0x7f6444169380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f6449d9b640 1 -- 172.21.15.186:0/3936275979 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440e0480 msgr2=0x7f644416a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f6449d9b640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440e0480 0x7f644416a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f6449d9b640 1 -- 172.21.15.186:0/3936275979 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6444047ce0 con 0x7f64440d9240 2024-03-28T15:01:37.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f644a59c640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f644404ffc0 0x7f6444169380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:37.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f6449d9b640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64440d9240 0x7f6444169f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6440001dc0 tx=0x7f644000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7fd91ffff640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd90806c600 0x7fd90806eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7fd91ffff640 1 -- 172.21.15.186:0/4201696495 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd9200adab0 con 0x7fd92806f230 2024-03-28T15:01:37.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7fd91ffff640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] conn(0x7fd908074800 0x7fd908076c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7fd91ffff640 1 -- 172.21.15.186:0/4201696495 --> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9080772d0 con 0x7fd908074800 2024-03-28T15:01:37.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7fd91ffff640 1 -- 172.21.15.186:0/4201696495 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd920077d70 con 0x7fd92806f230 2024-03-28T15:01:37.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7fd9267fc640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd90806c600 0x7fd90806eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7fd9277fe640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] conn(0x7fd908074800 0x7fd908076c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7fd9277fe640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] conn(0x7fd908074800 0x7fd908076c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f643affd640 1 -- 172.21.15.186:0/3936275979 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6440015d70 con 0x7f64440d9240 2024-03-28T15:01:37.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f643affd640 1 -- 172.21.15.186:0/3936275979 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64400050d0 con 0x7f64440d9240 2024-03-28T15:01:37.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7fabbb7fe640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faba406c600 0x7faba406eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faba8001030 con 0x7fabbc04c690 2024-03-28T15:01:37.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f643affd640 1 -- 172.21.15.186:0/3936275979 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64400054a0 con 0x7f64440d9240 2024-03-28T15:01:37.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f2103fff640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f20e806c630 0x7f20e806eaf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f20f0006e80 tx=0x7f20f0006dd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7fabc259c640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faba406c600 0x7faba406eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7fabbb7fe640 1 -- 172.21.15.186:0/4108234262 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fabc410f5c0 con 0x7fabbc04c690 2024-03-28T15:01:37.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.631+0000 7f8bbba15640 1 -- 172.21.15.186:0/535902944 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb415be40 con 0x7f8bb415ad70 2024-03-28T15:01:37.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.633+0000 7f8ba9ffb640 1 -- 172.21.15.186:0/535902944 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8ba4037070 con 0x7f8bb415ad70 2024-03-28T15:01:37.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f8ba9ffb640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8b9807d5c0 0x7f8b9807fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f8ba9ffb640 1 -- 172.21.15.186:0/535902944 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8ba4078ae0 con 0x7f8bb415ad70 2024-03-28T15:01:37.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7fabbb7fe640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] conn(0x7faba4074800 0x7faba4076c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f88feffd640 1 -- 172.21.15.186:0/2525547612 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f890808c070 con 0x7f89000e2e00 2024-03-28T15:01:37.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f88feffd640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f88ec06c600 0x7f88ec06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f88feffd640 1 -- 172.21.15.186:0/2525547612 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f89080f5be0 con 0x7f89000e2e00 2024-03-28T15:01:37.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f88feffd640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] conn(0x7f88ec074800 0x7f88ec076c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.634+0000 7f88feffd640 1 -- 172.21.15.186:0/2525547612 --> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f890806cb20 con 0x7f88ec074800 2024-03-28T15:01:37.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7fd9267fc640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd90806c600 0x7fd90806eac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd9281e39b0 tx=0x7fd910007a40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7f88feffd640 1 -- 172.21.15.186:0/2525547612 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f89080bfda0 con 0x7f89000e2e00 2024-03-28T15:01:37.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7f8bb97b1640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8b9807d5c0 0x7f8b9807fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7f6450c81640 1 -- 172.21.15.186:0/3936275979 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6444174c30 con 0x7f64440d9240 2024-03-28T15:01:37.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7f890e236640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f88ec06c600 0x7f88ec06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7fd91ffff640 1 -- 172.21.15.186:0/4201696495 <== osd.1 v2:172.21.15.186:6800/2980378029 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9080772d0 con 0x7fd908074800 2024-03-28T15:01:37.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7f890da35640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] conn(0x7f88ec074800 0x7f88ec076c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7fd7bed9d640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7ac06c580 0x7fd7ac06ea40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0036000 tx=0x7fd7b003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7f2108c0c640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] conn(0x7f20e80748b0 0x7f20e8076cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7fabc359e640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] conn(0x7faba4074800 0x7faba4076c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7fabc259c640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faba406c600 0x7faba406eac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fabbc04a0d0 tx=0x7fabb4009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7fabc359e640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] conn(0x7faba4074800 0x7faba4076c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7fabbb7fe640 1 -- 172.21.15.186:0/4108234262 --> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fabc4062df0 con 0x7faba4074800 2024-03-28T15:01:37.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7f6450c81640 1 -- 172.21.15.186:0/3936275979 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64441751f0 con 0x7f64440d9240 2024-03-28T15:01:37.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7f890e236640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f88ec06c600 0x7f88ec06eac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8904034000 tx=0x7f8904008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7f890da35640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] conn(0x7f88ec074800 0x7f88ec076c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7f8bb97b1640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8b9807d5c0 0x7f8b9807fa80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0005880 tx=0x7f8bb0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7f8bbba15640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] conn(0x7f8bb4151350 0x7f8bb415d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7f8bbba15640 1 -- 172.21.15.186:0/535902944 --> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8bb415dd70 con 0x7f8bb4151350 2024-03-28T15:01:37.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7fd7c504e640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] conn(0x7fd7c01f3ef0 0x7fd7c0148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7f8bb9fb2640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] conn(0x7f8bb4151350 0x7f8bb415d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.635+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 --> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7c00686d0 con 0x7fd7c01f3ef0 2024-03-28T15:01:37.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7fd7bf59e640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] conn(0x7fd7c01f3ef0 0x7fd7c0148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7f2108c0c640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] conn(0x7f20e80748b0 0x7f20e8076cd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.637+0000 7f6450c81640 1 -- 172.21.15.186:0/3936275979 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6444000cd0 con 0x7f64440d9240 2024-03-28T15:01:37.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.636+0000 7ff4b97fa640 1 -- 172.21.15.186:0/1470959126 <== osd.4 v2:172.21.15.191:6824/1698221714 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4ac020ec0 con 0x7ff4a8074730 2024-03-28T15:01:37.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.637+0000 7fabbb7fe640 1 -- 172.21.15.186:0/4108234262 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fabc40a5020 con 0x7fabbc04c690 2024-03-28T15:01:37.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.637+0000 7fd7bf59e640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] conn(0x7fd7c01f3ef0 0x7fd7c0148770 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.637+0000 7f88feffd640 1 -- 172.21.15.186:0/2525547612 <== osd.3 v2:172.21.15.186:6816/2463630609 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f890806cb20 con 0x7f88ec074800 2024-03-28T15:01:37.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.637+0000 7f2100ff9640 1 -- 172.21.15.186:0/942415693 <== osd.5 v2:172.21.15.191:6808/138503776 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f20f8001d30 con 0x7f20e80748b0 2024-03-28T15:01:37.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.637+0000 7f8bb9fb2640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] conn(0x7f8bb4151350 0x7f8bb415d830 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.637+0000 7fabbb7fe640 1 -- 172.21.15.186:0/4108234262 <== osd.0 v2:172.21.15.186:6824/2448652000 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fabc4062df0 con 0x7faba4074800 2024-03-28T15:01:37.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.637+0000 7f643affd640 1 -- 172.21.15.186:0/3936275979 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f64400035f0 con 0x7f64440d9240 2024-03-28T15:01:37.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.638+0000 7f8ba9ffb640 1 -- 172.21.15.186:0/535902944 <== osd.6 v2:172.21.15.191:6800/1108197747 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8bb415dd70 con 0x7f8bb4151350 2024-03-28T15:01:37.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.638+0000 7f643affd640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f642c06c7a0 0x7f642c06ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.638+0000 7f644a59c640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f642c06c7a0 0x7f642c06ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.638+0000 7fd7a77fe640 1 -- 172.21.15.186:0/3128938452 <== osd.7 v2:172.21.15.191:6816/562993402 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd7c00686d0 con 0x7fd7c01f3ef0 2024-03-28T15:01:37.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.639+0000 7f644a59c640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f642c06c7a0 0x7f642c06ec60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f643c036000 tx=0x7f643c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.639+0000 7f643affd640 1 -- 172.21.15.186:0/3936275979 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6440092c20 con 0x7f64440d9240 2024-03-28T15:01:37.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.640+0000 7f643affd640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] conn(0x7f642c0749a0 0x7f642c076dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.640+0000 7f643affd640 1 -- 172.21.15.186:0/3936275979 --> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f644001fe50 con 0x7f642c0749a0 2024-03-28T15:01:37.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.640+0000 7f643affd640 1 -- 172.21.15.186:0/3936275979 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f64400972a0 con 0x7f64440d9240 2024-03-28T15:01:37.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.640+0000 7f644ad9d640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] conn(0x7f642c0749a0 0x7f642c076dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.640+0000 7f644ad9d640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] conn(0x7f642c0749a0 0x7f642c076dc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.642+0000 7f643affd640 1 -- 172.21.15.186:0/3936275979 <== osd.2 v2:172.21.15.186:6808/2659582963 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f644001fe50 con 0x7f642c0749a0 2024-03-28T15:01:37.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.669+0000 7ff4c1c21640 1 -- 172.21.15.186:0/1470959126 --> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff4bc069500 con 0x7ff4a8074730 2024-03-28T15:01:37.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.669+0000 7f890f238640 1 -- 172.21.15.186:0/2525547612 --> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f890019f230 con 0x7f88ec074800 2024-03-28T15:01:37.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.669+0000 7f20ee7fc640 1 -- 172.21.15.186:0/942415693 --> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f20fc0daba0 con 0x7f20e80748b0 2024-03-28T15:01:37.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.669+0000 7ff4b97fa640 1 -- 172.21.15.186:0/1470959126 <== osd.4 v2:172.21.15.191:6824/1698221714 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff4bc069500 con 0x7ff4a8074730 2024-03-28T15:01:37.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.669+0000 7fd92d9f7640 1 -- 172.21.15.186:0/4201696495 --> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd90c002dd0 con 0x7fd908074800 2024-03-28T15:01:37.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.669+0000 7f2100ff9640 1 -- 172.21.15.186:0/942415693 <== osd.5 v2:172.21.15.191:6808/138503776 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f20fc0daba0 con 0x7f20e80748b0 2024-03-28T15:01:37.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7f88feffd640 1 -- 172.21.15.186:0/2525547612 <== osd.3 v2:172.21.15.186:6816/2463630609 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f890019f230 con 0x7f88ec074800 2024-03-28T15:01:37.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7ff4a2ffd640 1 -- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] conn(0x7ff4a8074730 msgr2=0x7ff4a8076b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7ff4a2ffd640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] conn(0x7ff4a8074730 0x7ff4a8076b50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7fd91ffff640 1 -- 172.21.15.186:0/4201696495 <== osd.1 v2:172.21.15.186:6800/2980378029 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd90c002dd0 con 0x7fd908074800 2024-03-28T15:01:37.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7f20ee7fc640 1 -- 172.21.15.186:0/942415693 >> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] conn(0x7f20e80748b0 msgr2=0x7f20e8076cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7ff4a2ffd640 1 -- 172.21.15.186:0/1470959126 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff4a806c530 msgr2=0x7ff4a806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7ff4a2ffd640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff4a806c530 0x7ff4a806e9f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff4b403b000 tx=0x7ff4b4036000 comp rx=0 tx=0).stop 2024-03-28T15:01:37.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7ff4a2ffd640 1 -- 172.21.15.186:0/1470959126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4bc0c7fc0 msgr2=0x7ff4bc164230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7f20ee7fc640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] conn(0x7f20e80748b0 0x7f20e8076cd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7ff4a2ffd640 1 --2- 172.21.15.186:0/1470959126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4bc0c7fc0 0x7ff4bc164230 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac00b3e0 tx=0x7ff4ac00b8a0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7ff4c1420640 1 -- 172.21.15.186:0/1470959126 reap_dead start 2024-03-28T15:01:37.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 --> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faba8002dd0 con 0x7faba4074800 2024-03-28T15:01:37.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7f20ee7fc640 1 -- 172.21.15.186:0/942415693 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f20e806c630 msgr2=0x7f20e806eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7f20ee7fc640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f20e806c630 0x7f20e806eaf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f20f0006e80 tx=0x7f20f0006dd0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7f20ee7fc640 1 -- 172.21.15.186:0/942415693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20fc0dd440 msgr2=0x7f20fc0e5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7f20ee7fc640 1 --2- 172.21.15.186:0/942415693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20fc0dd440 0x7f20fc0e5e40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f20f8000c00 tx=0x7f20f80046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7ff4a2ffd640 1 -- 172.21.15.186:0/1470959126 shutdown_connections 2024-03-28T15:01:37.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7ff4a2ffd640 1 -- 172.21.15.186:0/1470959126 >> 172.21.15.186:0/1470959126 conn(0x7ff4bc06b460 msgr2=0x7ff4bc0ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7ff4a2ffd640 1 -- 172.21.15.186:0/1470959126 shutdown_connections 2024-03-28T15:01:37.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7f2108c0c640 1 -- 172.21.15.186:0/942415693 reap_dead start 2024-03-28T15:01:37.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7f8bbba15640 1 -- 172.21.15.186:0/535902944 --> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8bb40686d0 con 0x7f8bb4151350 2024-03-28T15:01:37.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7fd92d9f7640 1 -- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] conn(0x7fd908074800 msgr2=0x7fd908076c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7fd92d9f7640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] conn(0x7fd908074800 0x7fd908076c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7ff4a2ffd640 1 -- 172.21.15.186:0/1470959126 wait complete. 2024-03-28T15:01:37.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7f890f238640 1 -- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] conn(0x7f88ec074800 msgr2=0x7f88ec076c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.670+0000 7f890f238640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] conn(0x7f88ec074800 0x7f88ec076c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fd92d9f7640 1 -- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd90806c600 msgr2=0x7fd90806eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fd92d9f7640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd90806c600 0x7fd90806eac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd9281e39b0 tx=0x7fd910007a40 comp rx=0 tx=0).stop 2024-03-28T15:01:37.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f890f238640 1 -- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f88ec06c600 msgr2=0x7f88ec06eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fabbb7fe640 1 -- 172.21.15.186:0/4108234262 <== osd.0 v2:172.21.15.186:6824/2448652000 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faba8002dd0 con 0x7faba4074800 2024-03-28T15:01:37.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f890f238640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f88ec06c600 0x7f88ec06eac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8904034000 tx=0x7f8904008030 comp rx=0 tx=0).stop 2024-03-28T15:01:37.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fd92d9f7640 1 -- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd92806f230 msgr2=0x7fd928148e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f890f238640 1 -- 172.21.15.186:0/2525547612 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000e2e00 msgr2=0x7f8900052370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f890f238640 1 --2- 172.21.15.186:0/2525547612 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000e2e00 0x7f8900052370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8908050b20 tx=0x7f89080691b0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fd92d9f7640 1 --2- 172.21.15.186:0/4201696495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd92806f230 0x7fd928148e80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd920000f80 tx=0x7fd920004a40 comp rx=0 tx=0).stop 2024-03-28T15:01:37.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fd9277fe640 1 -- 172.21.15.186:0/4201696495 reap_dead start 2024-03-28T15:01:37.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f890ea37640 1 -- 172.21.15.186:0/2525547612 reap_dead start 2024-03-28T15:01:37.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f890f238640 1 -- 172.21.15.186:0/2525547612 shutdown_connections 2024-03-28T15:01:37.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f8ba9ffb640 1 -- 172.21.15.186:0/535902944 <== osd.6 v2:172.21.15.191:6800/1108197747 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8bb40686d0 con 0x7f8bb4151350 2024-03-28T15:01:37.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f890f238640 1 -- 172.21.15.186:0/2525547612 >> 172.21.15.186:0/2525547612 conn(0x7f890000e690 msgr2=0x7f8900050290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fd92d9f7640 1 -- 172.21.15.186:0/4201696495 shutdown_connections 2024-03-28T15:01:37.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fd92d9f7640 1 -- 172.21.15.186:0/4201696495 >> 172.21.15.186:0/4201696495 conn(0x7fd92806c010 msgr2=0x7fd92814f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f890f238640 1 -- 172.21.15.186:0/2525547612 shutdown_connections 2024-03-28T15:01:37.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f890f238640 1 -- 172.21.15.186:0/2525547612 wait complete. 2024-03-28T15:01:37.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f20ee7fc640 1 -- 172.21.15.186:0/942415693 shutdown_connections 2024-03-28T15:01:37.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f20ee7fc640 1 -- 172.21.15.186:0/942415693 >> 172.21.15.186:0/942415693 conn(0x7f20fc050170 msgr2=0x7f20fc0d8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] conn(0x7faba4074800 msgr2=0x7faba4076c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fabc90f4640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] conn(0x7faba4074800 0x7faba4076c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 --> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd7c00cb4b0 con 0x7fd7c01f3ef0 2024-03-28T15:01:37.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f8bbba15640 1 -- 172.21.15.186:0/535902944 >> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] conn(0x7f8bb4151350 msgr2=0x7f8bb415d830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f8bbba15640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] conn(0x7f8bb4151350 0x7f8bb415d830 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7f20ee7fc640 1 -- 172.21.15.186:0/942415693 shutdown_connections 2024-03-28T15:01:37.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fd92d9f7640 1 -- 172.21.15.186:0/4201696495 shutdown_connections 2024-03-28T15:01:37.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.671+0000 7fd92d9f7640 1 -- 172.21.15.186:0/4201696495 wait complete. 2024-03-28T15:01:37.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7f20ee7fc640 1 -- 172.21.15.186:0/942415693 wait complete. 2024-03-28T15:01:37.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7f8bbba15640 1 -- 172.21.15.186:0/535902944 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8b9807d5c0 msgr2=0x7f8b9807fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7f8bbba15640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8b9807d5c0 0x7f8b9807fa80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0005880 tx=0x7f8bb0009040 comp rx=0 tx=0).stop 2024-03-28T15:01:37.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faba406c600 msgr2=0x7faba406eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7fabc90f4640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faba406c600 0x7faba406eac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fabbc04a0d0 tx=0x7fabb4009240 comp rx=0 tx=0).stop 2024-03-28T15:01:37.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fabbc04c690 msgr2=0x7fabbc171000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7fabc90f4640 1 --2- 172.21.15.186:0/4108234262 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fabbc04c690 0x7fabbc171000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fabc4069de0 tx=0x7fabc40691b0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7fabc359e640 1 -- 172.21.15.186:0/4108234262 reap_dead start 2024-03-28T15:01:37.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7fd7a77fe640 1 -- 172.21.15.186:0/3128938452 <== osd.7 v2:172.21.15.191:6816/562993402 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd7c00cb4b0 con 0x7fd7c01f3ef0 2024-03-28T15:01:37.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7f8bbba15640 1 -- 172.21.15.186:0/535902944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bb415ad70 msgr2=0x7f8bb40ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7f8bbba15640 1 --2- 172.21.15.186:0/535902944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bb415ad70 0x7f8bb40ccb90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40098c0 tx=0x7f8ba40046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7f8bb9fb2640 1 -- 172.21.15.186:0/535902944 reap_dead start 2024-03-28T15:01:37.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7f8bbba15640 1 -- 172.21.15.186:0/535902944 shutdown_connections 2024-03-28T15:01:37.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7f8bbba15640 1 -- 172.21.15.186:0/535902944 >> 172.21.15.186:0/535902944 conn(0x7f8bb406b880 msgr2=0x7f8bb4152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7f8bbba15640 1 -- 172.21.15.186:0/535902944 shutdown_connections 2024-03-28T15:01:37.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7f8bbba15640 1 -- 172.21.15.186:0/535902944 wait complete. 2024-03-28T15:01:37.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 shutdown_connections 2024-03-28T15:01:37.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 >> 172.21.15.186:0/4108234262 conn(0x7fabbc04dd60 msgr2=0x7fabbc0d3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 >> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] conn(0x7fd7c01f3ef0 msgr2=0x7fd7c0148770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7fd7c504e640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] conn(0x7fd7c01f3ef0 0x7fd7c0148770 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.672+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 shutdown_connections 2024-03-28T15:01:37.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.673+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7ac06c580 msgr2=0x7fd7ac06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.673+0000 7fd7c504e640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7ac06c580 0x7fd7ac06ea40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0036000 tx=0x7fd7b003b040 comp rx=0 tx=0).stop 2024-03-28T15:01:37.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.673+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c00cb6d0 msgr2=0x7fd7c0151440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.673+0000 7fd7c504e640 1 --2- 172.21.15.186:0/3128938452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c00cb6d0 0x7fd7c0151440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c3f0 tx=0x7fd7b400c8b0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.673+0000 7fd7bf59e640 1 -- 172.21.15.186:0/3128938452 reap_dead start 2024-03-28T15:01:37.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.673+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 shutdown_connections 2024-03-28T15:01:37.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.673+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 >> 172.21.15.186:0/3128938452 conn(0x7fd7c006fa00 msgr2=0x7fd7c0071170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.673+0000 7fabc90f4640 1 -- 172.21.15.186:0/4108234262 wait complete. 2024-03-28T15:01:37.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.673+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 shutdown_connections 2024-03-28T15:01:37.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.673+0000 7fd7c504e640 1 -- 172.21.15.186:0/3128938452 wait complete. 2024-03-28T15:01:37.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.674+0000 7f6450c81640 1 -- 172.21.15.186:0/3936275979 --> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f644400eff0 con 0x7f642c0749a0 2024-03-28T15:01:37.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.675+0000 7f643affd640 1 -- 172.21.15.186:0/3936275979 <== osd.2 v2:172.21.15.186:6808/2659582963 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f644400eff0 con 0x7f642c0749a0 2024-03-28T15:01:37.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.676+0000 7f6450c81640 1 -- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] conn(0x7f642c0749a0 msgr2=0x7f642c076dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.676+0000 7f6450c81640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] conn(0x7f642c0749a0 0x7f642c076dc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.676+0000 7f6450c81640 1 -- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f642c06c7a0 msgr2=0x7f642c06ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.677+0000 7f6450c81640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f642c06c7a0 0x7f642c06ec60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f643c036000 tx=0x7f643c008030 comp rx=0 tx=0).stop 2024-03-28T15:01:37.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.677+0000 7f6450c81640 1 -- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64440d9240 msgr2=0x7f6444169f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.677+0000 7f6450c81640 1 --2- 172.21.15.186:0/3936275979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64440d9240 0x7f6444169f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6440001dc0 tx=0x7f644000c7c0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.677+0000 7f644ad9d640 1 -- 172.21.15.186:0/3936275979 reap_dead start 2024-03-28T15:01:37.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.677+0000 7f6450c81640 1 -- 172.21.15.186:0/3936275979 shutdown_connections 2024-03-28T15:01:37.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.677+0000 7f6450c81640 1 -- 172.21.15.186:0/3936275979 >> 172.21.15.186:0/3936275979 conn(0x7f644400e150 msgr2=0x7f64440523c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.678+0000 7f6450c81640 1 -- 172.21.15.186:0/3936275979 shutdown_connections 2024-03-28T15:01:37.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.678+0000 7f6450c81640 1 -- 172.21.15.186:0/3936275979 wait complete. 2024-03-28T15:01:37.718 INFO:teuthology.orchestra.run.smithi186.stdout:55834574856 2024-03-28T15:01:37.718 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-28T15:01:37.719 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2024-03-28T15:01:37.719 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-28T15:01:37.720 INFO:teuthology.orchestra.run.smithi186.stdout:55834574856 2024-03-28T15:01:37.720 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-28T15:01:37.721 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2024-03-28T15:01:37.721 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-28T15:01:37.722 INFO:teuthology.orchestra.run.smithi186.stdout:55834574856 2024-03-28T15:01:37.722 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-28T15:01:37.723 INFO:teuthology.orchestra.run.smithi186.stdout:55834574856 2024-03-28T15:01:37.723 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-28T15:01:37.724 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2024-03-28T15:01:37.724 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-28T15:01:37.725 INFO:teuthology.orchestra.run.smithi186.stdout:55834574857 2024-03-28T15:01:37.725 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-28T15:01:37.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.929+0000 7f10801bf640 1 Processor -- start 2024-03-28T15:01:37.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.929+0000 7f10801bf640 1 -- start start 2024-03-28T15:01:37.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.929+0000 7f87588b1640 1 Processor -- start 2024-03-28T15:01:37.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.930+0000 7f87588b1640 1 -- start start 2024-03-28T15:01:37.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.930+0000 7f10801bf640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10700d5380 0x7f10700dc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.930+0000 7f10801bf640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10700dcbb0 0x7f10700de690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f10801bf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10700072e0 con 0x7f10700debd0 2024-03-28T15:01:37.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f10801bf640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f10700075c0 con 0x7f10700d5380 2024-03-28T15:01:37.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f10801bf640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f10700078a0 con 0x7f10700dcbb0 2024-03-28T15:01:37.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f107df5b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10700d5380 0x7f10700dc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f87588b1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8754158af0 0x7f87540c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f69a6f55640 1 Processor -- start 2024-03-28T15:01:37.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f107d75a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10700dcbb0 0x7f10700de690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f107d75a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10700dcbb0 0x7f10700de690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:56998/0 (socket says 172.21.15.186:56998) 2024-03-28T15:01:37.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f107d75a640 1 -- 172.21.15.186:0/3227831404 learned_addr learned my addr 172.21.15.186:0/3227831404 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f87537fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8754158af0 0x7f87540c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f87537fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8754158af0 0x7f87540c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41398/0 (socket says 172.21.15.186:41398) 2024-03-28T15:01:37.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f87537fe640 1 -- 172.21.15.186:0/399903992 learned_addr learned my addr 172.21.15.186:0/399903992 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f87588b1640 1 -- 172.21.15.186:0/399903992 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8754073a20 con 0x7f87540c6b60 2024-03-28T15:01:37.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f87588b1640 1 -- 172.21.15.186:0/399903992 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f875415ed60 con 0x7f875415f820 2024-03-28T15:01:37.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f87588b1640 1 -- 172.21.15.186:0/399903992 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f875415f390 con 0x7f8754158af0 2024-03-28T15:01:37.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f02a6fe2640 1 Processor -- start 2024-03-28T15:01:37.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f87527fc640 1 -- 172.21.15.186:0/399903992 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 804503923 0 0) 0x7f8754073a20 con 0x7f87540c6b60 2024-03-28T15:01:37.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f87527fc640 1 -- 172.21.15.186:0/399903992 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f873c003520 con 0x7f87540c6b60 2024-03-28T15:01:37.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f69a6f55640 1 -- start start 2024-03-28T15:01:37.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f01fe8a4640 1 Processor -- start 2024-03-28T15:01:37.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7fa14c95d640 1 Processor -- start 2024-03-28T15:01:37.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f02a6fe2640 1 -- start start 2024-03-28T15:01:37.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7fa14c95d640 1 -- start start 2024-03-28T15:01:37.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f01fe8a4640 1 -- start start 2024-03-28T15:01:37.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.931+0000 7f107d75a640 1 -- 172.21.15.186:0/3227831404 >> v2:172.21.15.186:3300/0 conn(0x7f10700d5380 msgr2=0x7f10700dc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f107d75a640 1 --2- 172.21.15.186:0/3227831404 >> v2:172.21.15.186:3300/0 conn(0x7f10700d5380 0x7f10700dc670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f107d75a640 1 -- 172.21.15.186:0/3227831404 >> v1:172.21.15.186:6789/0 conn(0x7f10700debd0 legacy=0x7f1070006210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f107d75a640 1 -- 172.21.15.186:0/3227831404 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1070007b80 con 0x7f10700dcbb0 2024-03-28T15:01:37.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7fc5d2038640 1 Processor -- start 2024-03-28T15:01:37.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f69a6f55640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f699804ba90 0x7f699804beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f69a6f55640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980e30c0 0x7f69980e3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f69a6f55640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6998046ba0 con 0x7f69980dd3f0 2024-03-28T15:01:37.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f69a6f55640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f6998045480 con 0x7f699804ba90 2024-03-28T15:01:37.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f69a6f55640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f6998046000 con 0x7f69980e30c0 2024-03-28T15:01:37.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.933+0000 7fc5d2038640 1 -- start start 2024-03-28T15:01:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.933+0000 7f02a6fe2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02980e6920 0x7f02980e33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01fe8a4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f01f00d55d0 0x7f01f004b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01fe8a4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01f004cf40 0x7f01f0045ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01fe8a4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f01f0045ff0 0x7f01f00de7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f69a4cf1640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f69980dd3f0 0x7f6998046490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:37484/0 (socket says 172.21.15.186:37484) 2024-03-28T15:01:37.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f69a4cf1640 1 -- 172.21.15.186:0/3109764307 learned_addr learned my addr 172.21.15.186:0/3109764307 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:37.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.933+0000 7fa14c95d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa148073c20 0x7fa148072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.933+0000 7fa146d9d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa148072bc0 0x7fa1480c1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:36152/0 (socket says 172.21.15.186:36152) 2024-03-28T15:01:37.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fa14659c640 1 --2- 172.21.15.186:0/1206503524 >> v2:172.21.15.191:3300/0 conn(0x7fa148073c20 0x7fa148072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fa146d9d640 1 -- 172.21.15.186:0/1206503524 learned_addr learned my addr 172.21.15.186:0/1206503524 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:37.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.933+0000 7f02a6fe2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f029804b9b0 con 0x7f029804c170 2024-03-28T15:01:37.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.933+0000 7f02a6fe2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0298007a20 con 0x7f0298006f30 2024-03-28T15:01:37.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.933+0000 7f02a6fe2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f029804bce0 con 0x7f02980e6920 2024-03-28T15:01:37.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.933+0000 7f029ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02980e6920 0x7f02980e33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f029ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02980e6920 0x7f02980e33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41404/0 (socket says 172.21.15.186:41404) 2024-03-28T15:01:37.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f029ffff640 1 -- 172.21.15.186:0/2403503498 learned_addr learned my addr 172.21.15.186:0/2403503498 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.933+0000 7f02a4d7e640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f0298006f30 0x7f0298007310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:36136/0 (socket says 172.21.15.186:36136) 2024-03-28T15:01:37.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f107d75a640 1 --2- 172.21.15.186:0/3227831404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10700dcbb0 0x7f10700de690 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1074009870 tx=0x7f107402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69fc232428df495d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01f7fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f01f00d55d0 0x7f01f004b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01f77fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01f004cf40 0x7f01f0045ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01f7fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f01f00d55d0 0x7f01f004b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58336/0 (socket says 172.21.15.186:58336) 2024-03-28T15:01:37.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fc5d2038640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc151870 0x7fc5cc157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fc5d2038640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc5cc06ca70 0x7fc5cc158210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fa14c95d640 1 -- 172.21.15.186:0/1206503524 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa14806eb10 con 0x7fa1480711f0 2024-03-28T15:01:37.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01f77fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01f004cf40 0x7f01f0045ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:57016/0 (socket says 172.21.15.186:57016) 2024-03-28T15:01:37.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01f7fff640 1 -- 172.21.15.186:0/4073836692 learned_addr learned my addr 172.21.15.186:0/4073836692 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f87527fc640 1 -- 172.21.15.186:0/399903992 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 260837969 0 0) 0x7f875415ed60 con 0x7f875415f820 2024-03-28T15:01:37.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f87527fc640 1 -- 172.21.15.186:0/399903992 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8754073a20 con 0x7f875415f820 2024-03-28T15:01:37.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f87527fc640 1 -- 172.21.15.186:0/399903992 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4212728875 0 0) 0x7f873c003520 con 0x7f87540c6b60 2024-03-28T15:01:37.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f87537fe640 1 -- 172.21.15.186:0/399903992 >> v1:172.21.15.191:6789/0 conn(0x7f875415f820 legacy=0x7f87540c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f87537fe640 1 -- 172.21.15.186:0/399903992 >> v1:172.21.15.186:6789/0 conn(0x7f87540c6b60 legacy=0x7f875406cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.932+0000 7f87537fe640 1 -- 172.21.15.186:0/399903992 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8754073ce0 con 0x7f8754158af0 2024-03-28T15:01:37.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f87537fe640 1 --2- 172.21.15.186:0/399903992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8754158af0 0x7f87540c7fc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f875415f510 tx=0x7f874c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c904ed5aa0e54a58 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fa14c95d640 1 -- 172.21.15.186:0/1206503524 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1481573c0 con 0x7fa148072bc0 2024-03-28T15:01:37.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fa14c95d640 1 -- 172.21.15.186:0/1206503524 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fa14806f620 con 0x7fa148073c20 2024-03-28T15:01:37.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01fe8a4640 1 -- 172.21.15.186:0/4073836692 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f01f00d59b0 con 0x7f01f0045ff0 2024-03-28T15:01:37.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01fe8a4640 1 -- 172.21.15.186:0/4073836692 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f01f004c390 con 0x7f01f00d55d0 2024-03-28T15:01:37.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fc5d2038640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc5cc0c7930 0x7fc5cc06fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01fe8a4640 1 -- 172.21.15.186:0/4073836692 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f01f004bf60 con 0x7f01f004cf40 2024-03-28T15:01:37.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f699ffff640 1 --2- 172.21.15.186:0/3109764307 >> v2:172.21.15.191:3300/0 conn(0x7f699804ba90 0x7f699804beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f107cf59640 1 -- 172.21.15.186:0/3227831404 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074034070 con 0x7f10700dcbb0 2024-03-28T15:01:37.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f107cf59640 1 -- 172.21.15.186:0/3227831404 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1074030840 con 0x7f10700dcbb0 2024-03-28T15:01:37.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f69a54f2640 1 --2- 172.21.15.186:0/3109764307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980e30c0 0x7f69980e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fc5d0835640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc5cc06ca70 0x7fc5cc158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fc5d2038640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc5cc073f20 con 0x7fc5cc0c7930 2024-03-28T15:01:37.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fc5d2038640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fc5cc0c8e40 con 0x7fc5cc06ca70 2024-03-28T15:01:37.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fc5d2038640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc5cc0733a0 con 0x7fc5cc151870 2024-03-28T15:01:37.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fc5d0835640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc5cc06ca70 0x7fc5cc158210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58346/0 (socket says 172.21.15.186:58346) 2024-03-28T15:01:37.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fc5d0835640 1 -- 172.21.15.186:0/3732093511 learned_addr learned my addr 172.21.15.186:0/3732093511 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fc5d1036640 1 --2- 172.21.15.186:0/3732093511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc151870 0x7fc5cc157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fc5d1837640 1 --2- 172.21.15.186:0/3732093511 >> v2:172.21.15.186:3300/0 conn(0x7fc5cc0c7930 0x7fc5cc06fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7fa145d9b640 1 -- 172.21.15.186:0/1206503524 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1215577825 0 0) 0x7fa1481573c0 con 0x7fa148072bc0 2024-03-28T15:01:37.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f01fce41640 1 --2- 172.21.15.186:0/4073836692 >> v2:172.21.15.186:3300/0 conn(0x7f01f0045ff0 0x7f01f00de7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.934+0000 7f029f7fe640 1 -- 172.21.15.186:0/2403503498 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 721429942 0 0) 0x7f029804b9b0 con 0x7f029804c170 2024-03-28T15:01:37.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f828ef89640 1 Processor -- start 2024-03-28T15:01:37.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7fc5d1036640 1 -- 172.21.15.186:0/3732093511 >> v2:172.21.15.191:3300/0 conn(0x7fc5cc06ca70 msgr2=0x7fc5cc158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7fc5d1036640 1 --2- 172.21.15.186:0/3732093511 >> v2:172.21.15.191:3300/0 conn(0x7fc5cc06ca70 0x7fc5cc158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7fc5d1036640 1 -- 172.21.15.186:0/3732093511 >> v2:172.21.15.186:3300/0 conn(0x7fc5cc0c7930 msgr2=0x7fc5cc06fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f029f7fe640 1 -- 172.21.15.186:0/2403503498 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0290003520 con 0x7f029804c170 2024-03-28T15:01:37.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f01f7fff640 1 -- 172.21.15.186:0/4073836692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01f004cf40 msgr2=0x7f01f0045ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f01f7fff640 1 --2- 172.21.15.186:0/4073836692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01f004cf40 0x7f01f0045ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f01f7fff640 1 -- 172.21.15.186:0/4073836692 >> v2:172.21.15.186:3300/0 conn(0x7f01f0045ff0 msgr2=0x7f01f00de7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f01f7fff640 1 --2- 172.21.15.186:0/4073836692 >> v2:172.21.15.186:3300/0 conn(0x7f01f0045ff0 0x7f01f00de7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f01f7fff640 1 -- 172.21.15.186:0/4073836692 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f0046480 con 0x7f01f00d55d0 2024-03-28T15:01:37.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f87527fc640 1 -- 172.21.15.186:0/399903992 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8748001670 con 0x7f8754158af0 2024-03-28T15:01:37.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f87527fc640 1 -- 172.21.15.186:0/399903992 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f874c02c7d0 con 0x7f8754158af0 2024-03-28T15:01:37.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f87527fc640 1 -- 172.21.15.186:0/399903992 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c02cba0 con 0x7f8754158af0 2024-03-28T15:01:37.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7fc5d1036640 1 --2- 172.21.15.186:0/3732093511 >> v2:172.21.15.186:3300/0 conn(0x7fc5cc0c7930 0x7fc5cc06fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7fc5d1036640 1 -- 172.21.15.186:0/3732093511 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5cc073680 con 0x7fc5cc151870 2024-03-28T15:01:37.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f69a54f2640 1 -- 172.21.15.186:0/3109764307 >> v2:172.21.15.191:3300/0 conn(0x7f699804ba90 msgr2=0x7f699804beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f69a54f2640 1 --2- 172.21.15.186:0/3109764307 >> v2:172.21.15.191:3300/0 conn(0x7f699804ba90 0x7f699804beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f69a54f2640 1 -- 172.21.15.186:0/3109764307 >> v1:172.21.15.186:6789/0 conn(0x7f69980dd3f0 legacy=0x7f6998046490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f69a54f2640 1 -- 172.21.15.186:0/3109764307 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69980dd7d0 con 0x7f69980e30c0 2024-03-28T15:01:37.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f029f7fe640 1 -- 172.21.15.186:0/2403503498 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1225029508 0 0) 0x7f0298007a20 con 0x7f0298006f30 2024-03-28T15:01:37.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f029f7fe640 1 -- 172.21.15.186:0/2403503498 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f029804b9b0 con 0x7f0298006f30 2024-03-28T15:01:37.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f10801bf640 1 -- 172.21.15.186:0/3227831404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10700dcbb0 msgr2=0x7f10700de690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f107cf59640 1 -- 172.21.15.186:0/3227831404 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074034070 con 0x7f10700dcbb0 2024-03-28T15:01:37.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f10801bf640 1 --2- 172.21.15.186:0/3227831404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10700dcbb0 0x7f10700de690 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1074009870 tx=0x7f107402b980 comp rx=0 tx=0).stop 2024-03-28T15:01:37.952 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f828ef89640 1 -- start start 2024-03-28T15:01:37.952 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7fa14659c640 1 -- 172.21.15.186:0/1206503524 >> v1:172.21.15.191:6789/0 conn(0x7fa148072bc0 legacy=0x7fa1480c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.952 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f87588b1640 1 -- 172.21.15.186:0/399903992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8754158af0 msgr2=0x7f87540c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.952 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7fa14659c640 1 -- 172.21.15.186:0/1206503524 >> v1:172.21.15.186:6789/0 conn(0x7fa1480711f0 legacy=0x7fa148161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.953 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7fa14659c640 1 -- 172.21.15.186:0/1206503524 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14806ed80 con 0x7fa148073c20 2024-03-28T15:01:37.953 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f87588b1640 1 --2- 172.21.15.186:0/399903992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8754158af0 0x7f87540c7fc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f875415f510 tx=0x7f874c02beb0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.953 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f10801bf640 1 -- 172.21.15.186:0/3227831404 shutdown_connections 2024-03-28T15:01:37.953 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f10801bf640 1 --2- 172.21.15.186:0/3227831404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10700dcbb0 0x7f10700de690 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.953 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f10801bf640 1 --2- 172.21.15.186:0/3227831404 >> v2:172.21.15.186:3300/0 conn(0x7f10700d5380 0x7f10700dc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.953 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f10801bf640 1 -- 172.21.15.186:0/3227831404 >> 172.21.15.186:0/3227831404 conn(0x7f10700509e0 msgr2=0x7f1070050df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f01f7fff640 1 --2- 172.21.15.186:0/4073836692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01f00d55d0 0x7f01f004b940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f01f004b070 tx=0x7f01ec02b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ed9493bbb49510d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f87588b1640 1 -- 172.21.15.186:0/399903992 shutdown_connections 2024-03-28T15:01:37.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f87588b1640 1 --2- 172.21.15.186:0/399903992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8754158af0 0x7f87540c7fc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7fc5d1036640 1 --2- 172.21.15.186:0/3732093511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc151870 0x7fc5cc157cd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc06ced0 tx=0x7fc5c802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17d78fc332e0f31f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f87588b1640 1 -- 172.21.15.186:0/399903992 >> 172.21.15.186:0/399903992 conn(0x7f875406b460 msgr2=0x7f875406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f69a54f2640 1 --2- 172.21.15.186:0/3109764307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980e30c0 0x7f69980e3530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f69a002d900 tx=0x7f69a00a7680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73251939c4a3b71f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7fa14659c640 1 --2- 172.21.15.186:0/1206503524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa148073c20 0x7fa148072fa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0097a0 tx=0x7fa13c02ba00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c90d836208324ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f029ffff640 1 -- 172.21.15.186:0/2403503498 >> v1:172.21.15.191:6789/0 conn(0x7f0298006f30 legacy=0x7f0298007310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f029ffff640 1 -- 172.21.15.186:0/2403503498 >> v1:172.21.15.186:6789/0 conn(0x7f029804c170 legacy=0x7f029804c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.935+0000 7f029ffff640 1 -- 172.21.15.186:0/2403503498 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02980e39f0 con 0x7f02980e6920 2024-03-28T15:01:37.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f01f6ffd640 1 -- 172.21.15.186:0/4073836692 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec035020 con 0x7f01f00d55d0 2024-03-28T15:01:37.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f01f6ffd640 1 -- 172.21.15.186:0/4073836692 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01ec02d040 con 0x7f01f00d55d0 2024-03-28T15:01:37.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f01f6ffd640 1 -- 172.21.15.186:0/4073836692 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec031a80 con 0x7f01f00d55d0 2024-03-28T15:01:37.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f87588b1640 1 -- 172.21.15.186:0/399903992 shutdown_connections 2024-03-28T15:01:37.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7fc5c3fff640 1 -- 172.21.15.186:0/3732093511 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80315a0 con 0x7fc5cc151870 2024-03-28T15:01:37.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f10801bf640 1 -- 172.21.15.186:0/3227831404 shutdown_connections 2024-03-28T15:01:37.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7fc5c3fff640 1 -- 172.21.15.186:0/3732093511 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5c8037070 con 0x7fc5cc151870 2024-03-28T15:01:37.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7fc5c3fff640 1 -- 172.21.15.186:0/3732093511 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8031960 con 0x7fc5cc151870 2024-03-28T15:01:37.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f699f7fe640 1 -- 172.21.15.186:0/3109764307 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a00ac7d0 con 0x7f69980e30c0 2024-03-28T15:01:37.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f699f7fe640 1 -- 172.21.15.186:0/3109764307 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69a00ac9b0 con 0x7f69980e30c0 2024-03-28T15:01:37.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f699f7fe640 1 -- 172.21.15.186:0/3109764307 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a00ace20 con 0x7f69980e30c0 2024-03-28T15:01:37.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.936+0000 7f87588b1640 1 -- 172.21.15.186:0/399903992 wait complete. 2024-03-28T15:01:37.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7fa145d9b640 1 -- 172.21.15.186:0/1206503524 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c02f820 con 0x7fa148073c20 2024-03-28T15:01:37.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f828ef89640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8288150980 0x7f8288156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7fa145d9b640 1 -- 172.21.15.186:0/1206503524 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa13c02f9c0 con 0x7fa148073c20 2024-03-28T15:01:37.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7fa145d9b640 1 -- 172.21.15.186:0/1206503524 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c02fdd0 con 0x7fa148073c20 2024-03-28T15:01:37.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f10801bf640 1 -- 172.21.15.186:0/3227831404 wait complete. 2024-03-28T15:01:37.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f01fe8a4640 1 -- 172.21.15.186:0/4073836692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01f00d55d0 msgr2=0x7f01f004b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f01fe8a4640 1 --2- 172.21.15.186:0/4073836692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01f00d55d0 0x7f01f004b940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f01f004b070 tx=0x7f01ec02b900 comp rx=0 tx=0).stop 2024-03-28T15:01:37.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f01fe8a4640 1 -- 172.21.15.186:0/4073836692 shutdown_connections 2024-03-28T15:01:37.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f01fe8a4640 1 --2- 172.21.15.186:0/4073836692 >> v2:172.21.15.186:3300/0 conn(0x7f01f0045ff0 0x7f01f00de7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f01fe8a4640 1 --2- 172.21.15.186:0/4073836692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01f004cf40 0x7f01f0045ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f01fe8a4640 1 --2- 172.21.15.186:0/4073836692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01f00d55d0 0x7f01f004b940 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f01fe8a4640 1 -- 172.21.15.186:0/4073836692 >> 172.21.15.186:0/4073836692 conn(0x7f01f004d4b0 msgr2=0x7f01f004d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f01fe8a4640 1 -- 172.21.15.186:0/4073836692 shutdown_connections 2024-03-28T15:01:37.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f828ef89640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f82880c68b0 0x7f828806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f828ef89640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82880c1400 con 0x7f828806dae0 2024-03-28T15:01:37.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f828ef89640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f828806cde0 con 0x7f8288150980 2024-03-28T15:01:37.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f828df87640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8288150980 0x7f8288156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f828e788640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f828806dae0 0x7f82880c7530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:36164/0 (socket says 172.21.15.186:36164) 2024-03-28T15:01:37.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f828e788640 1 -- 172.21.15.186:0/2630921012 learned_addr learned my addr 172.21.15.186:0/2630921012 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:37.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f828ef89640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f8288156d50 con 0x7f82880c68b0 2024-03-28T15:01:37.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f69a6f55640 1 -- 172.21.15.186:0/3109764307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980e30c0 msgr2=0x7f69980e3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f69a6f55640 1 --2- 172.21.15.186:0/3109764307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980e30c0 0x7f69980e3530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f69a002d900 tx=0x7f69a00a7680 comp rx=0 tx=0).stop 2024-03-28T15:01:37.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7fc5d2038640 1 -- 172.21.15.186:0/3732093511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc151870 msgr2=0x7fc5cc157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7fc5d2038640 1 --2- 172.21.15.186:0/3732093511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc151870 0x7fc5cc157cd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc06ced0 tx=0x7fc5c802b840 comp rx=0 tx=0).stop 2024-03-28T15:01:37.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f029ffff640 1 --2- 172.21.15.186:0/2403503498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02980e6920 0x7f02980e33b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0294009870 tx=0x7f029402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1088bf50c1d6f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f69a6f55640 1 -- 172.21.15.186:0/3109764307 shutdown_connections 2024-03-28T15:01:37.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.937+0000 7f69a6f55640 1 --2- 172.21.15.186:0/3109764307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980e30c0 0x7f69980e3530 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f69a6f55640 1 --2- 172.21.15.186:0/3109764307 >> v2:172.21.15.191:3300/0 conn(0x7f699804ba90 0x7f699804beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f69a6f55640 1 -- 172.21.15.186:0/3109764307 >> 172.21.15.186:0/3109764307 conn(0x7f69980501b0 msgr2=0x7f69980505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f029f7fe640 1 -- 172.21.15.186:0/2403503498 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c001690 con 0x7f02980e6920 2024-03-28T15:01:37.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f029f7fe640 1 -- 172.21.15.186:0/2403503498 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0294037070 con 0x7f02980e6920 2024-03-28T15:01:37.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728879696 0 0) 0x7f82880c1400 con 0x7f828806dae0 2024-03-28T15:01:37.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828d786640 1 --2- 172.21.15.186:0/2630921012 >> v2:172.21.15.191:3300/0 conn(0x7f82880c68b0 0x7f828806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f029f7fe640 1 -- 172.21.15.186:0/2403503498 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02940317b0 con 0x7f02980e6920 2024-03-28T15:01:37.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f01fe8a4640 1 -- 172.21.15.186:0/4073836692 wait complete. 2024-03-28T15:01:37.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7fc5d2038640 1 -- 172.21.15.186:0/3732093511 shutdown_connections 2024-03-28T15:01:37.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7fa14c95d640 1 -- 172.21.15.186:0/1206503524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa148073c20 msgr2=0x7fa148072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7fa14c95d640 1 --2- 172.21.15.186:0/1206503524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa148073c20 0x7fa148072fa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0097a0 tx=0x7fa13c02ba00 comp rx=0 tx=0).stop 2024-03-28T15:01:37.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7fc5d2038640 1 --2- 172.21.15.186:0/3732093511 >> v2:172.21.15.186:3300/0 conn(0x7fc5cc0c7930 0x7fc5cc06fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7fc5d2038640 1 --2- 172.21.15.186:0/3732093511 >> v2:172.21.15.191:3300/0 conn(0x7fc5cc06ca70 0x7fc5cc158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7fc5d2038640 1 --2- 172.21.15.186:0/3732093511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc151870 0x7fc5cc157cd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8274003520 con 0x7f828806dae0 2024-03-28T15:01:37.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7fc5d2038640 1 -- 172.21.15.186:0/3732093511 >> 172.21.15.186:0/3732093511 conn(0x7fc5cc06e9f0 msgr2=0x7fc5cc0ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f87588b1640 1 Processor -- start 2024-03-28T15:01:37.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f10801bf640 1 Processor -- start 2024-03-28T15:01:37.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1309571070 0 0) 0x7f8274003520 con 0x7f828806dae0 2024-03-28T15:01:37.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 >> v2:172.21.15.191:3300/0 conn(0x7f82880c68b0 msgr2=0x7f828806d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 --2- 172.21.15.186:0/2630921012 >> v2:172.21.15.191:3300/0 conn(0x7f82880c68b0 0x7f828806d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 >> v2:172.21.15.186:3300/0 conn(0x7f8288150980 msgr2=0x7f8288156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 --2- 172.21.15.186:0/2630921012 >> v2:172.21.15.186:3300/0 conn(0x7f8288150980 0x7f8288156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8288157010 con 0x7f828806dae0 2024-03-28T15:01:37.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f8280001a10 con 0x7f828806dae0 2024-03-28T15:01:37.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f69a6f55640 1 -- 172.21.15.186:0/3109764307 shutdown_connections 2024-03-28T15:01:37.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f10801bf640 1 -- start start 2024-03-28T15:01:37.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 >> v1:172.21.15.191:6789/0 conn(0x7f828806dae0 legacy=0x7f82880c7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7fc5d2038640 1 -- 172.21.15.186:0/3732093511 shutdown_connections 2024-03-28T15:01:37.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7fc5d2038640 1 -- 172.21.15.186:0/3732093511 wait complete. 2024-03-28T15:01:37.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8274003a00 0x7f8274003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8274004800 0x7f8274004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8274005210 0x7f827400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f828806cde0 con 0x7f8274003a00 2024-03-28T15:01:37.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8288156d50 con 0x7f8274004800 2024-03-28T15:01:37.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f827400c1e0 con 0x7f8274005210 2024-03-28T15:01:37.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f828df87640 1 --2- 172.21.15.186:0/2630921012 >> v2:172.21.15.186:3300/0 conn(0x7f8288150980 0x7f8288156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:37.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f828df87640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8274003a00 0x7f8274003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7fa14c95d640 1 -- 172.21.15.186:0/1206503524 shutdown_connections 2024-03-28T15:01:37.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7fa14c95d640 1 --2- 172.21.15.186:0/1206503524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa148073c20 0x7fa148072fa0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7fa14c95d640 1 -- 172.21.15.186:0/1206503524 >> 172.21.15.186:0/1206503524 conn(0x7fa14806b460 msgr2=0x7fa14806b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f02a6fe2640 1 -- 172.21.15.186:0/2403503498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02980e6920 msgr2=0x7f02980e33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f02a6fe2640 1 --2- 172.21.15.186:0/2403503498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02980e6920 0x7f02980e33b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0294009870 tx=0x7f029402b8b0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f828d786640 1 --2- 172.21.15.186:0/2630921012 >> v2:172.21.15.191:3300/0 conn(0x7f82880c68b0 0x7f828806d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:37.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f828e788640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8274005210 0x7f827400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f87588b1640 1 -- start start 2024-03-28T15:01:37.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f828d786640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8274004800 0x7f8274004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f02a6fe2640 1 -- 172.21.15.186:0/2403503498 shutdown_connections 2024-03-28T15:01:37.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f02a6fe2640 1 --2- 172.21.15.186:0/2403503498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02980e6920 0x7f02980e33b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f02a6fe2640 1 -- 172.21.15.186:0/2403503498 >> 172.21.15.186:0/2403503498 conn(0x7f02980061a0 msgr2=0x7f02980065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f01fe8a4640 1 Processor -- start 2024-03-28T15:01:37.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.938+0000 7f69a6f55640 1 -- 172.21.15.186:0/3109764307 wait complete. 2024-03-28T15:01:37.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7fa14c95d640 1 -- 172.21.15.186:0/1206503524 shutdown_connections 2024-03-28T15:01:37.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f10801bf640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10700d5380 0x7f10700ddb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f10801bf640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10700dcbb0 0x7f10700da620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f10801bf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10700debd0 0x7f10700dab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.939+0000 7f10801bf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f10700dcfb0 con 0x7f10700debd0 2024-03-28T15:01:37.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f10801bf640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1070008140 con 0x7f10700dcbb0 2024-03-28T15:01:37.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f10801bf640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f107004d1d0 con 0x7f10700d5380 2024-03-28T15:01:37.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7fa14c95d640 1 -- 172.21.15.186:0/1206503524 wait complete. 2024-03-28T15:01:37.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f87588b1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87540c6b60 0x7f875415dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f828df87640 1 -- 172.21.15.186:0/2630921012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8274005210 msgr2=0x7f827400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f828df87640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8274005210 0x7f827400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f87588b1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8754158af0 0x7f87541f5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f828df87640 1 -- 172.21.15.186:0/2630921012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8274004800 msgr2=0x7f8274004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:37.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f828df87640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8274004800 0x7f8274004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f828df87640 1 -- 172.21.15.186:0/2630921012 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8274003520 con 0x7f8274003a00 2024-03-28T15:01:37.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f107e75c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10700debd0 0x7f10700dab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f107e75c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10700debd0 0x7f10700dab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41440/0 (socket says 172.21.15.186:41440) 2024-03-28T15:01:37.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f107e75c640 1 -- 172.21.15.186:0/4125275962 learned_addr learned my addr 172.21.15.186:0/4125275962 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f87537fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87540c6b60 0x7f875415dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f828e788640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8274005210 0x7f827400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:37.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f01fe8a4640 1 -- start start 2024-03-28T15:01:37.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f828e788640 1 -- 172.21.15.186:0/2630921012 reap_dead start 2024-03-28T15:01:37.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f87537fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87540c6b60 0x7f875415dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:57032/0 (socket says 172.21.15.186:57032) 2024-03-28T15:01:37.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f69a6f55640 1 Processor -- start 2024-03-28T15:01:37.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f87537fe640 1 -- 172.21.15.186:0/2409399642 learned_addr learned my addr 172.21.15.186:0/2409399642 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f87588b1640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f875415f820 0x7f87540c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8754158ed0 con 0x7f8754158af0 2024-03-28T15:01:37.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f107e75c640 1 -- 172.21.15.186:0/4125275962 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10700d5380 msgr2=0x7f10700ddb20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T15:01:37.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f107e75c640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10700d5380 0x7f10700ddb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f107e75c640 1 -- 172.21.15.186:0/4125275962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10700dcbb0 msgr2=0x7f10700da620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T15:01:37.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f107e75c640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10700dcbb0 0x7f10700da620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f107e75c640 1 -- 172.21.15.186:0/4125275962 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1070178bf0 con 0x7f10700debd0 2024-03-28T15:01:37.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7fc5d2038640 1 Processor -- start 2024-03-28T15:01:37.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7fc5d2038640 1 -- start start 2024-03-28T15:01:37.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f875406e4d0 con 0x7f875415f820 2024-03-28T15:01:37.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f107e75c640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10700debd0 0x7f10700dab60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f107807be50 tx=0x7f10780691b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8754159050 con 0x7f87540c6b60 2024-03-28T15:01:37.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f8752ffd640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8754158af0 0x7f87541f5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f8753fff640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f875415f820 0x7f87540c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f87537fe640 1 -- 172.21.15.186:0/2409399642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f875415f820 msgr2=0x7f87540c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f87537fe640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f875415f820 0x7f87540c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f87537fe640 1 -- 172.21.15.186:0/2409399642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8754158af0 msgr2=0x7f87541f5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f87537fe640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8754158af0 0x7f87541f5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f87537fe640 1 -- 172.21.15.186:0/2409399642 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87541f8980 con 0x7f87540c6b60 2024-03-28T15:01:37.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f87537fe640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87540c6b60 0x7f875415dc20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f874c009f40 tx=0x7f874c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.982 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f01fe8a4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01f0045ff0 0x7f01f004a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.982 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f873bfff640 1 -- 172.21.15.186:0/2409399642 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c043020 con 0x7f87540c6b60 2024-03-28T15:01:37.982 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f01fe8a4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01f004cf40 0x7f01f00501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.982 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f828df87640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8274003a00 0x7f8274003e20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8284009c40 tx=0x7f8274005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729a19e1fbcbdf20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f873bfff640 1 -- 172.21.15.186:0/2409399642 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f874c02ce00 con 0x7f87540c6b60 2024-03-28T15:01:37.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f873bfff640 1 -- 172.21.15.186:0/2409399642 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c03e450 con 0x7f87540c6b60 2024-03-28T15:01:37.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f01fe8a4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01f00d55d0 0x7f01f004f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f875415e3e0 con 0x7f87540c6b60 2024-03-28T15:01:37.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.941+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f875415e9a0 con 0x7f87540c6b60 2024-03-28T15:01:37.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f01fe8a4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01f0007970 con 0x7f01f004cf40 2024-03-28T15:01:37.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f01fe8a4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f01f00082a0 con 0x7f01f0045ff0 2024-03-28T15:01:37.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f01fe8a4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f01f0007c50 con 0x7f01f00d55d0 2024-03-28T15:01:37.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f01f77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01f004cf40 0x7f01f00501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f01f77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01f004cf40 0x7f01f00501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41466/0 (socket says 172.21.15.186:41466) 2024-03-28T15:01:37.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f01f77fe640 1 -- 172.21.15.186:0/1969134704 learned_addr learned my addr 172.21.15.186:0/1969134704 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f01f7fff640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01f0045ff0 0x7f01f004a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284009840 con 0x7f8274003a00 2024-03-28T15:01:37.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8284004020 con 0x7f8274003a00 2024-03-28T15:01:37.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f828cf85640 1 -- 172.21.15.186:0/2630921012 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284004490 con 0x7f8274003a00 2024-03-28T15:01:37.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.940+0000 7f69a6f55640 1 -- start start 2024-03-28T15:01:37.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f69a6f55640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f699804ba90 0x7f699804ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f69a6f55640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69980dd3f0 0x7f6998184e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f69a6f55640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980e30c0 0x7f69981853d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f69a6f55640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f69980459b0 con 0x7f69980dd3f0 2024-03-28T15:01:37.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f69a6f55640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f69980456d0 con 0x7f69980e30c0 2024-03-28T15:01:37.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f69a6f55640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f6998045c90 con 0x7f699804ba90 2024-03-28T15:01:37.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f699ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69980dd3f0 0x7f6998184e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f699ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69980dd3f0 0x7f6998184e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41482/0 (socket says 172.21.15.186:41482) 2024-03-28T15:01:37.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f699ffff640 1 -- 172.21.15.186:0/4025529521 learned_addr learned my addr 172.21.15.186:0/4025529521 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f69a4cf1640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f699804ba90 0x7f699804ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d2038640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5cc06ca70 0x7fc5cc13d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d2038640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc0c7930 0x7fc5cc155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d1036640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5cc06ca70 0x7fc5cc13d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d1036640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5cc06ca70 0x7fc5cc13d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41494/0 (socket says 172.21.15.186:41494) 2024-03-28T15:01:37.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d0835640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc0c7930 0x7fc5cc155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d2038640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc151870 0x7fc5cc142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d0835640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc0c7930 0x7fc5cc155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:57082/0 (socket says 172.21.15.186:57082) 2024-03-28T15:01:37.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d2038640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc1577b0 con 0x7fc5cc06ca70 2024-03-28T15:01:37.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d2038640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc073c40 con 0x7fc5cc151870 2024-03-28T15:01:37.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d2038640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc5cc151c50 con 0x7fc5cc0c7930 2024-03-28T15:01:37.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d1837640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc151870 0x7fc5cc142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d1837640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc151870 0x7fc5cc142ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58412/0 (socket says 172.21.15.186:58412) 2024-03-28T15:01:37.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fc5d1036640 1 -- 172.21.15.186:0/3761237367 learned_addr learned my addr 172.21.15.186:0/3761237367 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f02a6fe2640 1 -- 172.21.15.186:0/2403503498 shutdown_connections 2024-03-28T15:01:37.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f02a6fe2640 1 -- 172.21.15.186:0/2403503498 wait complete. 2024-03-28T15:01:37.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f106e7fc640 1 -- 172.21.15.186:0/4125275962 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107806c9c0 con 0x7f10700debd0 2024-03-28T15:01:37.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f106e7fc640 1 -- 172.21.15.186:0/4125275962 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f107806cb60 con 0x7f10700debd0 2024-03-28T15:01:37.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f01f7fff640 1 -- 172.21.15.186:0/1969134704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01f00d55d0 msgr2=0x7f01f004f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f01f7fff640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01f00d55d0 0x7f01f004f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f828ef89640 1 -- 172.21.15.186:0/2630921012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8274003a00 msgr2=0x7f8274003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f01f7fff640 1 -- 172.21.15.186:0/1969134704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01f004cf40 msgr2=0x7f01f00501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f01f7fff640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01f004cf40 0x7f01f00501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f01f7fff640 1 -- 172.21.15.186:0/1969134704 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f004a920 con 0x7f01f0045ff0 2024-03-28T15:01:37.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f828ef89640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8274003a00 0x7f8274003e20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8284009c40 tx=0x7f8274005830 comp rx=0 tx=0).stop 2024-03-28T15:01:37.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7f106e7fc640 1 -- 172.21.15.186:0/4125275962 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078094400 con 0x7f10700debd0 2024-03-28T15:01:37.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fc5d1837640 1 -- 172.21.15.186:0/3761237367 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc0c7930 msgr2=0x7fc5cc155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f828ef89640 1 -- 172.21.15.186:0/2630921012 shutdown_connections 2024-03-28T15:01:37.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f828ef89640 1 --2- 172.21.15.186:0/2630921012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8274003a00 0x7f8274003e20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fa14c95d640 1 Processor -- start 2024-03-28T15:01:37.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.942+0000 7fa14c95d640 1 -- start start 2024-03-28T15:01:37.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f10801bf640 1 -- 172.21.15.186:0/4125275962 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1070178ed0 con 0x7f10700debd0 2024-03-28T15:01:37.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f828ef89640 1 -- 172.21.15.186:0/2630921012 >> 172.21.15.186:0/2630921012 conn(0x7f828806b460 msgr2=0x7f828806b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:37.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fa14c95d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1480711f0 0x7fa14814cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f10801bf640 1 -- 172.21.15.186:0/4125275962 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10700d96d0 con 0x7f10700debd0 2024-03-28T15:01:37.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f699ffff640 1 -- 172.21.15.186:0/4025529521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f699804ba90 msgr2=0x7f699804ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f699ffff640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f699804ba90 0x7f699804ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f699ffff640 1 -- 172.21.15.186:0/4025529521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980e30c0 msgr2=0x7f69981853d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T15:01:37.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f699ffff640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980e30c0 0x7f69981853d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f699ffff640 1 -- 172.21.15.186:0/4025529521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69980dbf70 con 0x7f69980dd3f0 2024-03-28T15:01:37.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f828ef89640 1 -- 172.21.15.186:0/2630921012 shutdown_connections 2024-03-28T15:01:37.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fa14c95d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa148072bc0 0x7fa1481e2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f828ef89640 1 -- 172.21.15.186:0/2630921012 wait complete. 2024-03-28T15:01:37.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7f01f7fff640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01f0045ff0 0x7f01f004a3e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f01ec033000 tx=0x7f01ec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fc5d1837640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc0c7930 0x7fc5cc155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fc5d1837640 1 -- 172.21.15.186:0/3761237367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5cc06ca70 msgr2=0x7fc5cc13d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7f699ffff640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69980dd3f0 0x7f6998184e90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f699400e870 tx=0x7f699400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.945+0000 7f105bfff640 1 -- 172.21.15.186:0/4125275962 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1070051630 con 0x7f10700debd0 2024-03-28T15:01:37.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7f01f4ff9640 1 -- 172.21.15.186:0/1969134704 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec03ac60 con 0x7f01f0045ff0 2024-03-28T15:01:37.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7f01fe8a4640 1 -- 172.21.15.186:0/1969134704 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f00d7870 con 0x7f01f0045ff0 2024-03-28T15:01:37.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7f01fe8a4640 1 -- 172.21.15.186:0/1969134704 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f0049900 con 0x7f01f0045ff0 2024-03-28T15:01:37.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f01e27fc640 1 -- 172.21.15.186:0/1969134704 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d8005290 con 0x7f01f0045ff0 2024-03-28T15:01:37.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.947+0000 7f01f4ff9640 1 -- 172.21.15.186:0/1969134704 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01ec03ae20 con 0x7f01f0045ff0 2024-03-28T15:01:37.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.945+0000 7f828ef89640 1 Processor -- start 2024-03-28T15:01:37.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f828ef89640 1 -- start start 2024-03-28T15:01:37.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.947+0000 7f828ef89640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8274003a00 0x7f828814e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.947+0000 7f828ef89640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82880c5e10 0x7f82881ef440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8754073eb0 con 0x7f87540c6b60 2024-03-28T15:01:37.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fc5d1837640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5cc06ca70 0x7fc5cc13d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fc5d1837640 1 -- 172.21.15.186:0/3761237367 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5cc154bd0 con 0x7fc5cc151870 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fc5d1837640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc151870 0x7fc5cc142ea0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4007d90 tx=0x7fc5c400e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fc5c1ffb640 1 -- 172.21.15.186:0/3761237367 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c400edd0 con 0x7fc5cc151870 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7fc5c1ffb640 1 -- 172.21.15.186:0/3761237367 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5c4003990 con 0x7fc5cc151870 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7fc5c1ffb640 1 -- 172.21.15.186:0/3761237367 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4003d80 con 0x7fc5cc151870 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.945+0000 7fc5d2038640 1 -- 172.21.15.186:0/3761237367 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5cc154330 con 0x7fc5cc151870 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.945+0000 7fc5d2038640 1 -- 172.21.15.186:0/3761237367 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5cc1548f0 con 0x7fc5cc151870 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.947+0000 7fc5c1ffb640 1 -- 172.21.15.186:0/3761237367 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc5c401b020 con 0x7fc5cc151870 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.947+0000 7fc5c1ffb640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc5b006c530 0x7fc5b006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.947+0000 7fc5c1ffb640 1 -- 172.21.15.186:0/3761237367 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc5c408b810 con 0x7fc5cc151870 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fa14c95d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa148073c20 0x7fa14806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fa14659c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa148072bc0 0x7fa1481e2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fa14659c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa148072bc0 0x7fa1481e2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:57084/0 (socket says 172.21.15.186:57084) 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.943+0000 7fa14659c640 1 -- 172.21.15.186:0/3157342913 learned_addr learned my addr 172.21.15.186:0/3157342913 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1480c1640 con 0x7fa1480711f0 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7fa14759e640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa148073c20 0x7fa14806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa14806f340 con 0x7fa148073c20 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa1480c7ca0 con 0x7fa148072bc0 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7fa146d9d640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1480711f0 0x7fa14814cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7fa14659c640 1 -- 172.21.15.186:0/3157342913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa148073c20 msgr2=0x7fa14806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7fa14659c640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa148073c20 0x7fa14806cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7fa14659c640 1 -- 172.21.15.186:0/3157342913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1480711f0 msgr2=0x7fa14814cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.945+0000 7fa14659c640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1480711f0 0x7fa14814cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.945+0000 7fa14659c640 1 -- 172.21.15.186:0/3157342913 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1481573c0 con 0x7fa148072bc0 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.945+0000 7fa146d9d640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1480711f0 0x7fa14814cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.945+0000 7fa14659c640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa148072bc0 0x7fa1481e2940 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa13c02c010 tx=0x7fa13c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:37.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.945+0000 7fa12f7fe640 1 -- 172.21.15.186:0/3157342913 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c042480 con 0x7fa148072bc0 2024-03-28T15:01:37.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa14814d1e0 con 0x7fa148072bc0 2024-03-28T15:01:37.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1481f4bc0 con 0x7fa148072bc0 2024-03-28T15:01:37.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7f02a6fe2640 1 Processor -- start 2024-03-28T15:01:37.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7f02a6fe2640 1 -- start start 2024-03-28T15:01:37.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f02a6fe2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0298006f30 0x7f029817ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f02a6fe2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f029804c170 0x7f02980e5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f02a4d7e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0298006f30 0x7f029817ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f02a4d7e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0298006f30 0x7f029817ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58420/0 (socket says 172.21.15.186:58420) 2024-03-28T15:01:37.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f02a4d7e640 1 -- 172.21.15.186:0/653420452 learned_addr learned my addr 172.21.15.186:0/653420452 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:37.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f02a6fe2640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02980e6920 0x7f02980ddc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f02980e4290 con 0x7f029804c170 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f02980e3fb0 con 0x7f0298006f30 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f02980dcfe0 con 0x7f02980e6920 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.947+0000 7f02a557f640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02980e6920 0x7f02980ddc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f02a557f640 1 -- 172.21.15.186:0/653420452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0298006f30 msgr2=0x7f029817ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f02a557f640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0298006f30 0x7f029817ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f02a557f640 1 -- 172.21.15.186:0/653420452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f029804c170 msgr2=0x7f02980e5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f02a557f640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f029804c170 0x7f02980e5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f02a557f640 1 -- 172.21.15.186:0/653420452 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0298178c10 con 0x7f02980e6920 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7f699d7fa640 1 -- 172.21.15.186:0/4025529521 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6994017070 con 0x7f69980dd3f0 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7f699d7fa640 1 -- 172.21.15.186:0/4025529521 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f699400c8e0 con 0x7f69980dd3f0 2024-03-28T15:01:37.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.944+0000 7f699d7fa640 1 -- 172.21.15.186:0/4025529521 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699400ccb0 con 0x7f69980dd3f0 2024-03-28T15:01:38.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.945+0000 7f69a6f55640 1 -- 172.21.15.186:0/4025529521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69980d9040 con 0x7f69980dd3f0 2024-03-28T15:01:38.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.945+0000 7f69a6f55640 1 -- 172.21.15.186:0/4025529521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69980d95c0 con 0x7f69980dd3f0 2024-03-28T15:01:38.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.947+0000 7f699d7fa640 1 -- 172.21.15.186:0/4025529521 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f699401a070 con 0x7f69980dd3f0 2024-03-28T15:01:38.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f69a6f55640 1 -- 172.21.15.186:0/4025529521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6998005b80 con 0x7f69980dd3f0 2024-03-28T15:01:38.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f02a557f640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02980e6920 0x7f02980ddc70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c780 tx=0x7f028c00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:38.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f699d7fa640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f698c06c630 0x7f698c06eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:38.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7fc5d1036640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc5b006c530 0x7fc5b006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:38.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f699d7fa640 1 -- 172.21.15.186:0/4025529521 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f699408b830 con 0x7f69980dd3f0 2024-03-28T15:01:38.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7fa12f7fe640 1 -- 172.21.15.186:0/3157342913 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa13c047070 con 0x7fa148072bc0 2024-03-28T15:01:38.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7fa12f7fe640 1 -- 172.21.15.186:0/3157342913 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c042860 con 0x7fa148072bc0 2024-03-28T15:01:38.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f029d7fa640 1 -- 172.21.15.186:0/653420452 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c017070 con 0x7f02980e6920 2024-03-28T15:01:38.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f029d7fa640 1 -- 172.21.15.186:0/653420452 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f028c005290 con 0x7f02980e6920 2024-03-28T15:01:38.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f029d7fa640 1 -- 172.21.15.186:0/653420452 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c00a880 con 0x7f02980e6920 2024-03-28T15:01:38.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0298179990 con 0x7f02980e6920 2024-03-28T15:01:38.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0298178eb0 con 0x7f02980e6920 2024-03-28T15:01:38.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f873bfff640 1 -- 172.21.15.186:0/2409399642 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f874c046020 con 0x7f87540c6b60 2024-03-28T15:01:38.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f873bfff640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f874006c530 0x7f874006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:38.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f873bfff640 1 -- 172.21.15.186:0/2409399642 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f874c0ad970 con 0x7f87540c6b60 2024-03-28T15:01:38.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f8752ffd640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f874006c530 0x7f874006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:38.003 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7fa12f7fe640 1 -- 172.21.15.186:0/3157342913 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa13c040050 con 0x7fa148072bc0 2024-03-28T15:01:38.003 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f01f4ff9640 1 -- 172.21.15.186:0/1969134704 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec044630 con 0x7f01f0045ff0 2024-03-28T15:01:38.003 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f01f4ff9640 1 -- 172.21.15.186:0/1969134704 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f01ec04b060 con 0x7f01f0045ff0 2024-03-28T15:01:38.003 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f01f4ff9640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f01dc06c630 0x7f01dc06eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:38.003 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f01f77fe640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f01dc06c630 0x7f01dc06eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:38.003 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f01f4ff9640 1 -- 172.21.15.186:0/1969134704 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f01ec0ad3e0 con 0x7f01f0045ff0 2024-03-28T15:01:38.004 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f69a4cf1640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f698c06c630 0x7f698c06eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:38.004 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828ef89640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82881ef980 0x7f8288146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:38.004 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f828df87640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8274003a00 0x7f828814e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:38.004 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828df87640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8274003a00 0x7f828814e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:58426/0 (socket says 172.21.15.186:58426) 2024-03-28T15:01:38.004 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7f828d786640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82880c5e10 0x7f82881ef440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:38.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f01f77fe640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f01dc06c630 0x7f01dc06eaf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f01f00c8280 tx=0x7f01e4012040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:38.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828d786640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82880c5e10 0x7f82881ef440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41514/0 (socket says 172.21.15.186:41514) 2024-03-28T15:01:38.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.949+0000 7fc5d2038640 1 -- 172.21.15.186:0/3761237367 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5cc075d40 con 0x7fc5cc151870 2024-03-28T15:01:38.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828df87640 1 -- 172.21.15.186:0/876841086 learned_addr learned my addr 172.21.15.186:0/876841086 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:38.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7fc5d1036640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc5b006c530 0x7fc5b006e9f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009560 tx=0x7fc5c803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:38.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f828806d080 con 0x7f82880c5e10 2024-03-28T15:01:38.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f82881575d0 con 0x7f8274003a00 2024-03-28T15:01:38.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8288150d60 con 0x7f82881ef980 2024-03-28T15:01:38.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828e788640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82881ef980 0x7f8288146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:38.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.946+0000 7f106e7fc640 1 -- 172.21.15.186:0/4125275962 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f10780945c0 con 0x7f10700debd0 2024-03-28T15:01:38.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.947+0000 7f106e7fc640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f105c06c600 0x7f105c06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:38.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.947+0000 7f106e7fc640 1 -- 172.21.15.186:0/4125275962 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1078106c80 con 0x7f10700debd0 2024-03-28T15:01:38.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.947+0000 7f107df5b640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f105c06c600 0x7f105c06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:38.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.948+0000 7f107df5b640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f105c06c600 0x7f105c06eac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1068009870 tx=0x7f1068008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:38.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f69a4cf1640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f698c06c630 0x7f698c06eaf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6990005880 tx=0x7f6990009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:38.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f8752ffd640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f874006c530 0x7f874006e9f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8748001af0 tx=0x7f8748001a80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:38.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7fa12f7fe640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa13406c660 0x7fa13406eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:38.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7fa146d9d640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa13406c660 0x7fa13406eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:38.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828e788640 1 -- 172.21.15.186:0/876841086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8274003a00 msgr2=0x7f828814e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.009 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828e788640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8274003a00 0x7f828814e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.009 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828e788640 1 -- 172.21.15.186:0/876841086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82880c5e10 msgr2=0x7f82881ef440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.009 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828e788640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82880c5e10 0x7f82881ef440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.009 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7f828e788640 1 -- 172.21.15.186:0/876841086 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82881efe00 con 0x7f82881ef980 2024-03-28T15:01:38.009 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.951+0000 7f828e788640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82881ef980 0x7f8288146670 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f828806e3c0 tx=0x7f82800079d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:38.009 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.950+0000 7fa12f7fe640 1 -- 172.21.15.186:0/3157342913 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa13c0ae3f0 con 0x7fa148072bc0 2024-03-28T15:01:38.009 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.951+0000 7fa146d9d640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa13406c660 0x7fa13406eb20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa1380075a0 tx=0x7fa138000f10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:38.009 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.951+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa128005290 con 0x7fa148072bc0 2024-03-28T15:01:38.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.951+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02980057e0 con 0x7f02980e6920 2024-03-28T15:01:38.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.953+0000 7f01f4ff9640 1 -- 172.21.15.186:0/1969134704 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f01ec03f070 con 0x7f01f0045ff0 2024-03-28T15:01:38.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.953+0000 7f106e7fc640 1 -- 172.21.15.186:0/4125275962 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f1070051630 con 0x7f10700debd0 2024-03-28T15:01:38.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.954+0000 7f873bfff640 1 -- 172.21.15.186:0/2409399642 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8754073eb0 con 0x7f87540c6b60 2024-03-28T15:01:38.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.954+0000 7f827e7fc640 1 -- 172.21.15.186:0/876841086 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828000c040 con 0x7f82881ef980 2024-03-28T15:01:38.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.954+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f828814acc0 con 0x7f82881ef980 2024-03-28T15:01:38.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.954+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f828814b280 con 0x7f82881ef980 2024-03-28T15:01:38.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.955+0000 7f029d7fa640 1 -- 172.21.15.186:0/653420452 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f028c01a070 con 0x7f02980e6920 2024-03-28T15:01:38.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.955+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f828806e460 con 0x7f82881ef980 2024-03-28T15:01:38.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.955+0000 7f029d7fa640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f028406c5b0 0x7f028406ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:38.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.955+0000 7f02a4d7e640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f028406c5b0 0x7f028406ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:38.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.956+0000 7f02a4d7e640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f028406c5b0 0x7f028406ea70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f02a0064b70 tx=0x7f02a004fec0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:38.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.957+0000 7fc5c1ffb640 1 -- 172.21.15.186:0/3761237367 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc5c40596b0 con 0x7fc5cc151870 2024-03-28T15:01:38.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.958+0000 7f827e7fc640 1 -- 172.21.15.186:0/876841086 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8280018070 con 0x7f82881ef980 2024-03-28T15:01:38.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.959+0000 7f827e7fc640 1 -- 172.21.15.186:0/876841086 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280014750 con 0x7f82881ef980 2024-03-28T15:01:38.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.959+0000 7f029d7fa640 1 -- 172.21.15.186:0/653420452 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f028c0570a0 con 0x7f02980e6920 2024-03-28T15:01:38.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.959+0000 7fa12f7fe640 1 -- 172.21.15.186:0/3157342913 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa13c033030 con 0x7fa148072bc0 2024-03-28T15:01:38.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.961+0000 7f699d7fa640 1 -- 172.21.15.186:0/4025529521 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6994059e30 con 0x7f69980dd3f0 2024-03-28T15:01:38.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.964+0000 7f029d7fa640 1 -- 172.21.15.186:0/653420452 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f028c021020 con 0x7f02980e6920 2024-03-28T15:01:38.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.965+0000 7f827e7fc640 1 -- 172.21.15.186:0/876841086 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f828000f050 con 0x7f82881ef980 2024-03-28T15:01:38.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.965+0000 7f827e7fc640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f826c06c560 0x7f826c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:38.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.966+0000 7f827e7fc640 1 -- 172.21.15.186:0/876841086 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f82800906e0 con 0x7f82881ef980 2024-03-28T15:01:38.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.966+0000 7f828df87640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f826c06c560 0x7f826c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:38.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.967+0000 7f828df87640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f826c06c560 0x7f826c06ea20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8284006f00 tx=0x7f8284030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:38.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:37.970+0000 7f827e7fc640 1 -- 172.21.15.186:0/876841086 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f828806e460 con 0x7f82881ef980 2024-03-28T15:01:38.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.376+0000 7f105bfff640 1 -- 172.21.15.186:0/4125275962 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f10700057b0 con 0x7f10700debd0 2024-03-28T15:01:38.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.376+0000 7fc5d2038640 1 -- 172.21.15.186:0/3761237367 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fc5cc0728f0 con 0x7fc5cc151870 2024-03-28T15:01:38.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.377+0000 7f106e7fc640 1 -- 172.21.15.186:0/4125275962 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f107810d080 con 0x7f10700debd0 2024-03-28T15:01:38.377 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2024-03-28T15:01:38.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.377+0000 7fc5c1ffb640 1 -- 172.21.15.186:0/3761237367 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc5c405d9b0 con 0x7fc5cc151870 2024-03-28T15:01:38.378 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2024-03-28T15:01:38.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.380+0000 7f10801bf640 1 -- 172.21.15.186:0/4125275962 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f105c06c600 msgr2=0x7f105c06eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.380+0000 7f10801bf640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f105c06c600 0x7f105c06eac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1068009870 tx=0x7f1068008040 comp rx=0 tx=0).stop 2024-03-28T15:01:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.380+0000 7f10801bf640 1 -- 172.21.15.186:0/4125275962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10700debd0 msgr2=0x7f10700dab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.380+0000 7f10801bf640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10700debd0 0x7f10700dab60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f107807be50 tx=0x7f10780691b0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.380+0000 7f01e27fc640 1 -- 172.21.15.186:0/1969134704 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f01d8005580 con 0x7f01f0045ff0 2024-03-28T15:01:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.380+0000 7fc5af7fe640 1 -- 172.21.15.186:0/3761237367 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc5b006c530 msgr2=0x7fc5b006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.380+0000 7fc5af7fe640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc5b006c530 0x7fc5b006e9f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009560 tx=0x7fc5c803d040 comp rx=0 tx=0).stop 2024-03-28T15:01:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.380+0000 7fc5af7fe640 1 -- 172.21.15.186:0/3761237367 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc151870 msgr2=0x7fc5cc142ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.380+0000 7fc5af7fe640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc151870 0x7fc5cc142ea0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4007d90 tx=0x7fc5c400e4b0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.380+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa128005580 con 0x7fa148072bc0 2024-03-28T15:01:38.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7fc5af7fe640 1 -- 172.21.15.186:0/3761237367 shutdown_connections 2024-03-28T15:01:38.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7fc5af7fe640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc5b006c530 0x7fc5b006e9f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7fc5af7fe640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc151870 0x7fc5cc142ea0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7fc5af7fe640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5cc0c7930 0x7fc5cc155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7f10801bf640 1 -- 172.21.15.186:0/4125275962 shutdown_connections 2024-03-28T15:01:38.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7fc5af7fe640 1 --2- 172.21.15.186:0/3761237367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5cc06ca70 0x7fc5cc13d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7f10801bf640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f105c06c600 0x7f105c06eac0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7fc5af7fe640 1 -- 172.21.15.186:0/3761237367 >> 172.21.15.186:0/3761237367 conn(0x7fc5cc06e9f0 msgr2=0x7fc5cc0c8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:38.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7f10801bf640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10700debd0 0x7f10700dab60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7f10801bf640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10700dcbb0 0x7f10700da620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7f10801bf640 1 --2- 172.21.15.186:0/4125275962 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10700d5380 0x7f10700ddb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7f10801bf640 1 -- 172.21.15.186:0/4125275962 >> 172.21.15.186:0/4125275962 conn(0x7f10700509e0 msgr2=0x7f107004c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7fc5af7fe640 1 -- 172.21.15.186:0/3761237367 shutdown_connections 2024-03-28T15:01:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7fc5af7fe640 1 -- 172.21.15.186:0/3761237367 wait complete. 2024-03-28T15:01:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7f10801bf640 1 -- 172.21.15.186:0/4125275962 shutdown_connections 2024-03-28T15:01:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7f10801bf640 1 -- 172.21.15.186:0/4125275962 wait complete. 2024-03-28T15:01:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7fa12f7fe640 1 -- 172.21.15.186:0/3157342913 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa13c07ca80 con 0x7fa148072bc0 2024-03-28T15:01:38.386 INFO:teuthology.orchestra.run.smithi186.stdout:55834574854 2024-03-28T15:01:38.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.381+0000 7f01f4ff9640 1 -- 172.21.15.186:0/1969134704 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f01ec07ba70 con 0x7f01f0045ff0 2024-03-28T15:01:38.386 INFO:teuthology.orchestra.run.smithi186.stdout:55834574854 2024-03-28T15:01:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.384+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f87541f4d50 con 0x7f87540c6b60 2024-03-28T15:01:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.384+0000 7f69a6f55640 1 -- 172.21.15.186:0/4025529521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f69980336c0 con 0x7f69980dd3f0 2024-03-28T15:01:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.384+0000 7f01e27fc640 1 -- 172.21.15.186:0/1969134704 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f01dc06c630 msgr2=0x7f01dc06eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.384+0000 7f01e27fc640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f01dc06c630 0x7f01dc06eaf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f01f00c8280 tx=0x7f01e4012040 comp rx=0 tx=0).stop 2024-03-28T15:01:38.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.384+0000 7f873bfff640 1 -- 172.21.15.186:0/2409399642 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f874c04d030 con 0x7f87540c6b60 2024-03-28T15:01:38.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.384+0000 7f01e27fc640 1 -- 172.21.15.186:0/1969134704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01f0045ff0 msgr2=0x7f01f004a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.384+0000 7f01e27fc640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01f0045ff0 0x7f01f004a3e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f01ec033000 tx=0x7f01ec0046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7f699d7fa640 1 -- 172.21.15.186:0/4025529521 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6994059730 con 0x7f69980dd3f0 2024-03-28T15:01:38.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa13406c660 msgr2=0x7fa13406eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.389 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2024-03-28T15:01:38.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7fa14c95d640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa13406c660 0x7fa13406eb20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa1380075a0 tx=0x7fa138000f10 comp rx=0 tx=0).stop 2024-03-28T15:01:38.389 INFO:teuthology.orchestra.run.smithi186.stdout:55834574854 2024-03-28T15:01:38.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa148072bc0 msgr2=0x7fa1481e2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7fa14c95d640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa148072bc0 0x7fa1481e2940 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa13c02c010 tx=0x7fa13c0046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 shutdown_connections 2024-03-28T15:01:38.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7fa14c95d640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa13406c660 0x7fa13406eb20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7fa14c95d640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa148073c20 0x7fa14806cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7fa14c95d640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa148072bc0 0x7fa1481e2940 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7fa14c95d640 1 --2- 172.21.15.186:0/3157342913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1480711f0 0x7fa14814cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 >> 172.21.15.186:0/3157342913 conn(0x7fa14806b460 msgr2=0x7fa1480ca520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:38.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7f01e27fc640 1 -- 172.21.15.186:0/1969134704 shutdown_connections 2024-03-28T15:01:38.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7f01e27fc640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f01dc06c630 0x7f01dc06eaf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7f01e27fc640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01f00d55d0 0x7f01f004f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7f01e27fc640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01f004cf40 0x7f01f00501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7f01e27fc640 1 --2- 172.21.15.186:0/1969134704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01f0045ff0 0x7f01f004a3e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.385+0000 7f01e27fc640 1 -- 172.21.15.186:0/1969134704 >> 172.21.15.186:0/1969134704 conn(0x7f01f004d4b0 msgr2=0x7f01f00d2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:38.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.386+0000 7f01e27fc640 1 -- 172.21.15.186:0/1969134704 shutdown_connections 2024-03-28T15:01:38.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.386+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 shutdown_connections 2024-03-28T15:01:38.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.386+0000 7fa14c95d640 1 -- 172.21.15.186:0/3157342913 wait complete. 2024-03-28T15:01:38.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.386+0000 7f01e27fc640 1 -- 172.21.15.186:0/1969134704 wait complete. 2024-03-28T15:01:38.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.388+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f029803a020 con 0x7f02980e6920 2024-03-28T15:01:38.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.388+0000 7f69a6f55640 1 -- 172.21.15.186:0/4025529521 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f698c06c630 msgr2=0x7f698c06eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.388+0000 7f69a6f55640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f698c06c630 0x7f698c06eaf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6990005880 tx=0x7f6990009040 comp rx=0 tx=0).stop 2024-03-28T15:01:38.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f69a6f55640 1 -- 172.21.15.186:0/4025529521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69980dd3f0 msgr2=0x7f6998184e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f69a6f55640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69980dd3f0 0x7f6998184e90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f699400e870 tx=0x7f699400ed30 comp rx=0 tx=0).stop 2024-03-28T15:01:38.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.388+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f874006c530 msgr2=0x7f874006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.388+0000 7f87588b1640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f874006c530 0x7f874006e9f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8748001af0 tx=0x7f8748001a80 comp rx=0 tx=0).stop 2024-03-28T15:01:38.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87540c6b60 msgr2=0x7f875415dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f87588b1640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87540c6b60 0x7f875415dc20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f874c009f40 tx=0x7f874c004a40 comp rx=0 tx=0).stop 2024-03-28T15:01:38.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 shutdown_connections 2024-03-28T15:01:38.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f87588b1640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f874006c530 0x7f874006e9f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f87588b1640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f875415f820 0x7f87540c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f87588b1640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8754158af0 0x7f87541f5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f87588b1640 1 --2- 172.21.15.186:0/2409399642 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87540c6b60 0x7f875415dc20 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f874c009f40 tx=0x7f874c004a40 comp rx=0 tx=0).stop 2024-03-28T15:01:38.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 >> 172.21.15.186:0/2409399642 conn(0x7f875406b460 msgr2=0x7f87540cadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:38.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f69a6f55640 1 -- 172.21.15.186:0/4025529521 shutdown_connections 2024-03-28T15:01:38.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f029d7fa640 1 -- 172.21.15.186:0/653420452 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f028c059000 con 0x7f02980e6920 2024-03-28T15:01:38.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f69a6f55640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f698c06c630 0x7f698c06eaf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f69a6f55640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980e30c0 0x7f69981853d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f69a6f55640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69980dd3f0 0x7f6998184e90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f69a6f55640 1 --2- 172.21.15.186:0/4025529521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f699804ba90 0x7f699804ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f69a6f55640 1 -- 172.21.15.186:0/4025529521 >> 172.21.15.186:0/4025529521 conn(0x7f69980501b0 msgr2=0x7f69980e1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:38.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f69a6f55640 1 -- 172.21.15.186:0/4025529521 shutdown_connections 2024-03-28T15:01:38.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f69a6f55640 1 -- 172.21.15.186:0/4025529521 wait complete. 2024-03-28T15:01:38.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 shutdown_connections 2024-03-28T15:01:38.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.389+0000 7f87588b1640 1 -- 172.21.15.186:0/2409399642 wait complete. 2024-03-28T15:01:38.396 INFO:teuthology.orchestra.run.smithi186.stdout:55834574856 2024-03-28T15:01:38.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.391+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f82880b1a00 con 0x7f82881ef980 2024-03-28T15:01:38.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.391+0000 7f827e7fc640 1 -- 172.21.15.186:0/876841086 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f82881efe00 con 0x7f82881ef980 2024-03-28T15:01:38.397 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2024-03-28T15:01:38.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.393+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f028406c5b0 msgr2=0x7f028406ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.393+0000 7f02a6fe2640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f028406c5b0 0x7f028406ea70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f02a0064b70 tx=0x7f02a004fec0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.393+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02980e6920 msgr2=0x7f02980ddc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.393+0000 7f02a6fe2640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02980e6920 0x7f02980ddc70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c780 tx=0x7f028c00cc40 comp rx=0 tx=0).stop 2024-03-28T15:01:38.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.394+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 shutdown_connections 2024-03-28T15:01:38.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.394+0000 7f02a6fe2640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f028406c5b0 0x7f028406ea70 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f02a0064b70 tx=0x7f02a004fec0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.394+0000 7f02a6fe2640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02980e6920 0x7f02980ddc70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.394+0000 7f02a6fe2640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f029804c170 0x7f02980e5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.394+0000 7f02a6fe2640 1 --2- 172.21.15.186:0/653420452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0298006f30 0x7f029817ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.394+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 >> 172.21.15.186:0/653420452 conn(0x7f02980061a0 msgr2=0x7f02980e6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:38.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.394+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 shutdown_connections 2024-03-28T15:01:38.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.395+0000 7f02a6fe2640 1 -- 172.21.15.186:0/653420452 wait complete. 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.396+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f826c06c560 msgr2=0x7f826c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.396+0000 7f828ef89640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f826c06c560 0x7f826c06ea20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8284006f00 tx=0x7f8284030000 comp rx=0 tx=0).stop 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.396+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82881ef980 msgr2=0x7f8288146670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.396+0000 7f828ef89640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82881ef980 0x7f8288146670 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f828806e3c0 tx=0x7f82800079d0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.396+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 shutdown_connections 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.396+0000 7f828ef89640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f826c06c560 0x7f826c06ea20 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8284006f00 tx=0x7f8284030000 comp rx=0 tx=0).stop 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.396+0000 7f828ef89640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82881ef980 0x7f8288146670 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.396+0000 7f828ef89640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82880c5e10 0x7f82881ef440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.397+0000 7f828ef89640 1 --2- 172.21.15.186:0/876841086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8274003a00 0x7f828814e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.397+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 >> 172.21.15.186:0/876841086 conn(0x7f828806b460 msgr2=0x7f828806b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.398+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 shutdown_connections 2024-03-28T15:01:38.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:38.398+0000 7f828ef89640 1 -- 172.21.15.186:0/876841086 wait complete. 2024-03-28T15:01:38.410 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2024-03-28T15:01:38.410 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2024-03-28T15:01:38.413 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2024-03-28T15:01:38.414 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2024-03-28T15:01:38.415 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2024-03-28T15:01:38.416 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2024-03-28T15:01:38.418 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.5 2024-03-28T15:01:38.419 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.2 2024-03-28T15:01:39.410 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-28T15:01:39.411 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-28T15:01:39.414 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-28T15:01:39.414 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-28T15:01:39.416 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-28T15:01:39.416 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-28T15:01:39.422 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-28T15:01:39.423 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-28T15:01:39.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.562+0000 7fd3b2447640 1 Processor -- start 2024-03-28T15:01:39.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.562+0000 7fd3b2447640 1 -- start start 2024-03-28T15:01:39.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fd3b2447640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3ac1507f0 0x7fd3ac0c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fd3b2447640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd3ac073c50 0x7fd3ac0c1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fd3b2447640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3ac0c6cd0 con 0x7fd3ac0c7f90 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fd3b2447640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fd3ac06e610 con 0x7fd3ac073c50 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fd3b2447640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd3ac06ecf0 con 0x7fd3ac1507f0 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fd3b09e4640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd3ac0c7f90 0x7fd3ac159d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51332/0 (socket says 172.21.15.186:51332) 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fd3b09e4640 1 -- 172.21.15.186:0/693265601 learned_addr learned my addr 172.21.15.186:0/693265601 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fd3ab7fe640 1 --2- 172.21.15.186:0/693265601 >> v2:172.21.15.191:3300/0 conn(0x7fd3ac073c50 0x7fd3ac0c1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 903125130 0 0) 0x7fd3ac0c6cd0 con 0x7fd3ac0c7f90 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd394007930 con 0x7fd3ac0c7f90 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 604129162 0 0) 0x7fd394007930 con 0x7fd3ac0c7f90 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3ac1507f0 msgr2=0x7fd3ac0c0f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T15:01:39.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 --2- 172.21.15.186:0/693265601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3ac1507f0 0x7fd3ac0c0f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.562+0000 7fdc265d6640 1 Processor -- start 2024-03-28T15:01:39.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.563+0000 7fdc265d6640 1 -- start start 2024-03-28T15:01:39.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fdc265d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc20164ee0 0x7fdc20168120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fdc265d6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdc2006f210 0x7fdc200c0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fdc265d6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc200c6b70 con 0x7fdc2006ee30 2024-03-28T15:01:39.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fdc265d6640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdc2006e4f0 con 0x7fdc2006f210 2024-03-28T15:01:39.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fdc265d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fdc200c6920 con 0x7fdc20164ee0 2024-03-28T15:01:39.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fdc24b73640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdc2006f210 0x7fdc200c0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fdc24b73640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdc2006f210 0x7fdc200c0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48244/0 (socket says 172.21.15.186:48244) 2024-03-28T15:01:39.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fdc24b73640 1 -- 172.21.15.186:0/2314262649 learned_addr learned my addr 172.21.15.186:0/2314262649 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.564+0000 7fdc1f7fe640 1 --2- 172.21.15.186:0/2314262649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc20164ee0 0x7fdc20168120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 >> v2:172.21.15.191:3300/0 conn(0x7fd3ac073c50 msgr2=0x7fd3ac0c1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 --2- 172.21.15.186:0/693265601 >> v2:172.21.15.191:3300/0 conn(0x7fd3ac073c50 0x7fd3ac0c1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.566+0000 7fdc1f7fe640 1 -- 172.21.15.186:0/2314262649 >> v2:172.21.15.186:3300/0 conn(0x7fdc2006f210 msgr2=0x7fdc200c0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac06f550 con 0x7fd3ac0c7f90 2024-03-28T15:01:39.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7fd3a0004e80 con 0x7fd3ac0c7f90 2024-03-28T15:01:39.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 >> v1:172.21.15.186:6789/0 conn(0x7fd3ac0c7f90 legacy=0x7fd3ac159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 --2- 172.21.15.186:0/693265601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd394007dc0 0x7fd3940081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 --2- 172.21.15.186:0/693265601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd394008c10 0x7fd394009090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 --2- 172.21.15.186:0/693265601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3940095d0 0x7fd39400fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac06e610 con 0x7fd3940095d0 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac06ecf0 con 0x7fd394008c10 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd3940104c0 con 0x7fd394007dc0 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3ab7fe640 1 --2- 172.21.15.186:0/693265601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd394008c10 0x7fd394009090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.565+0000 7fd3b09e4640 1 --2- 172.21.15.186:0/693265601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3940095d0 0x7fd39400fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.566+0000 7fd3b09e4640 1 -- 172.21.15.186:0/693265601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd394007dc0 msgr2=0x7fd3940081e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.566+0000 7fd3b09e4640 1 --2- 172.21.15.186:0/693265601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd394007dc0 0x7fd3940081e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.566+0000 7fd3b09e4640 1 -- 172.21.15.186:0/693265601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd394008c10 msgr2=0x7fd394009090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.566+0000 7fd3b09e4640 1 --2- 172.21.15.186:0/693265601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd394008c10 0x7fd394009090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.566+0000 7fd3b09e4640 1 -- 172.21.15.186:0/693265601 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac06f550 con 0x7fd3940095d0 2024-03-28T15:01:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.567+0000 7fdc1effd640 1 -- 172.21.15.186:0/2314262649 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2052785188 0 0) 0x7fdc200c6b70 con 0x7fdc2006ee30 2024-03-28T15:01:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.567+0000 7fdc1f7fe640 1 --2- 172.21.15.186:0/2314262649 >> v2:172.21.15.186:3300/0 conn(0x7fdc2006f210 0x7fdc200c0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.567+0000 7fdc1f7fe640 1 -- 172.21.15.186:0/2314262649 >> v1:172.21.15.191:6789/0 conn(0x7fdc2006ee30 legacy=0x7fdc201647d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.567+0000 7fdc1f7fe640 1 -- 172.21.15.186:0/2314262649 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc200c6e50 con 0x7fdc20164ee0 2024-03-28T15:01:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.567+0000 7fdc24b73640 1 --2- 172.21.15.186:0/2314262649 >> v2:172.21.15.186:3300/0 conn(0x7fdc2006f210 0x7fdc200c0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.567+0000 7fd3b09e4640 1 --2- 172.21.15.186:0/693265601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3940095d0 0x7fd39400fdd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0001b90 tx=0x7fd394009c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecd2f0ca6671842 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.567+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0049070 con 0x7fd3940095d0 2024-03-28T15:01:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.567+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a0009040 con 0x7fd3940095d0 2024-03-28T15:01:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.567+0000 7fd3aaffd640 1 -- 172.21.15.186:0/693265601 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0049070 con 0x7fd3940095d0 2024-03-28T15:01:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.568+0000 7fdc1f7fe640 1 --2- 172.21.15.186:0/2314262649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc20164ee0 0x7fdc20168120 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc140097a0 tx=0x7fdc1402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd4e471ec7083b55 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.568+0000 7fd3b2447640 1 -- 172.21.15.186:0/693265601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3940095d0 msgr2=0x7fd39400fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.569+0000 7fdc1effd640 1 -- 172.21.15.186:0/2314262649 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14034070 con 0x7fdc20164ee0 2024-03-28T15:01:39.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.568+0000 7fd3b2447640 1 --2- 172.21.15.186:0/693265601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3940095d0 0x7fd39400fdd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0001b90 tx=0x7fd394009c40 comp rx=0 tx=0).stop 2024-03-28T15:01:39.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.569+0000 7fdc1effd640 1 -- 172.21.15.186:0/2314262649 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc14030610 con 0x7fdc20164ee0 2024-03-28T15:01:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.568+0000 7fd3b09e4640 1 -- 172.21.15.186:0/693265601 reap_dead start 2024-03-28T15:01:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.568+0000 7fd3b2447640 1 -- 172.21.15.186:0/693265601 shutdown_connections 2024-03-28T15:01:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.568+0000 7fd3b2447640 1 -- 172.21.15.186:0/693265601 >> 172.21.15.186:0/693265601 conn(0x7fd3ac06c010 msgr2=0x7fd3ac0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.569+0000 7fd3b2447640 1 -- 172.21.15.186:0/693265601 shutdown_connections 2024-03-28T15:01:39.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.570+0000 7fdc265d6640 1 -- 172.21.15.186:0/2314262649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc20164ee0 msgr2=0x7fdc20168120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.570+0000 7fdc265d6640 1 --2- 172.21.15.186:0/2314262649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc20164ee0 0x7fdc20168120 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc140097a0 tx=0x7fdc1402b960 comp rx=0 tx=0).stop 2024-03-28T15:01:39.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.570+0000 7fdc265d6640 1 -- 172.21.15.186:0/2314262649 shutdown_connections 2024-03-28T15:01:39.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.570+0000 7fdc265d6640 1 --2- 172.21.15.186:0/2314262649 >> v2:172.21.15.186:3300/0 conn(0x7fdc2006f210 0x7fdc200c0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.570+0000 7fdc265d6640 1 --2- 172.21.15.186:0/2314262649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc20164ee0 0x7fdc20168120 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc140097a0 tx=0x7fdc1402b960 comp rx=0 tx=0).stop 2024-03-28T15:01:39.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.570+0000 7fdc265d6640 1 -- 172.21.15.186:0/2314262649 >> 172.21.15.186:0/2314262649 conn(0x7fdc2006c010 msgr2=0x7fdc200cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:39.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.570+0000 7fdc265d6640 1 -- 172.21.15.186:0/2314262649 shutdown_connections 2024-03-28T15:01:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.569+0000 7fd3b2447640 1 -- 172.21.15.186:0/693265601 wait complete. 2024-03-28T15:01:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.571+0000 7fd3b2447640 1 Processor -- start 2024-03-28T15:01:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.571+0000 7fd3b2447640 1 -- start start 2024-03-28T15:01:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.572+0000 7fd3b2447640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac1efa90 0x7fd3ac148710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.572+0000 7fd3b2447640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3ac1ed720 0x7fd3ac1efe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.573+0000 7fd3abfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac1efa90 0x7fd3ac148710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.573+0000 7fd3abfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac1efa90 0x7fd3ac148710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48264/0 (socket says 172.21.15.186:48264) 2024-03-28T15:01:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.573+0000 7fd3abfff640 1 -- 172.21.15.186:0/642371857 learned_addr learned my addr 172.21.15.186:0/642371857 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.573+0000 7fd3b2447640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3ac1f03b0 0x7fd3ac145990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.573+0000 7fd3ab7fe640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3ac1ed720 0x7fd3ac1efe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.573+0000 7fd3b2447640 1 -- 172.21.15.186:0/642371857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac0c19f0 con 0x7fd3ac1efa90 2024-03-28T15:01:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fd3b2447640 1 -- 172.21.15.186:0/642371857 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac06f270 con 0x7fd3ac1f03b0 2024-03-28T15:01:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fd3b2447640 1 -- 172.21.15.186:0/642371857 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd3ac150bd0 con 0x7fd3ac1ed720 2024-03-28T15:01:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.570+0000 7fdc265d6640 1 -- 172.21.15.186:0/2314262649 wait complete. 2024-03-28T15:01:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.572+0000 7fdc265d6640 1 Processor -- start 2024-03-28T15:01:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.572+0000 7fdc265d6640 1 -- start start 2024-03-28T15:01:39.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.573+0000 7fdc265d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc2006ee30 0x7fdc201667e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc265d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc2006f210 0x7fdc201fddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc265d6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc20169e00 0x7fdc20200fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fd3b09e4640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3ac1f03b0 0x7fd3ac145990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc24b73640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc20169e00 0x7fdc20200fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc24b73640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc20169e00 0x7fdc20200fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48274/0 (socket says 172.21.15.186:48274) 2024-03-28T15:01:39.581 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fd3ab7fe640 1 -- 172.21.15.186:0/642371857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3ac1f03b0 msgr2=0x7fd3ac145990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.581 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fd3ab7fe640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3ac1f03b0 0x7fd3ac145990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.581 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fd3ab7fe640 1 -- 172.21.15.186:0/642371857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac1efa90 msgr2=0x7fd3ac148710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.581 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fd3ab7fe640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac1efa90 0x7fd3ac148710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.581 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fd3ab7fe640 1 -- 172.21.15.186:0/642371857 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac156b10 con 0x7fd3ac1ed720 2024-03-28T15:01:39.582 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc1ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc2006ee30 0x7fdc201667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.582 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc265d6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc200c7410 con 0x7fdc20169e00 2024-03-28T15:01:39.582 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc265d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdc200c7780 con 0x7fdc2006ee30 2024-03-28T15:01:39.582 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc265d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fdc200c85d0 con 0x7fdc2006f210 2024-03-28T15:01:39.582 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc24b73640 1 -- 172.21.15.186:0/4263877904 learned_addr learned my addr 172.21.15.186:0/4263877904 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.582 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc1ffff640 1 -- 172.21.15.186:0/4263877904 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc2006f210 msgr2=0x7fdc201fddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc1ffff640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc2006f210 0x7fdc201fddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc1ffff640 1 -- 172.21.15.186:0/4263877904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc20169e00 msgr2=0x7fdc20200fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc1ffff640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc20169e00 0x7fdc20200fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc1ffff640 1 -- 172.21.15.186:0/4263877904 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc20200b80 con 0x7fdc2006ee30 2024-03-28T15:01:39.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.574+0000 7fdc24b73640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc20169e00 0x7fdc20200fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:39.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.575+0000 7fdc1ffff640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc2006ee30 0x7fdc201667e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800c750 tx=0x7fdc1800cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.575+0000 7fd3ab7fe640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3ac1ed720 0x7fd3ac1efe70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4004710 tx=0x7fd3a400a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.575+0000 7fdc1cff9640 1 -- 172.21.15.186:0/4263877904 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1800ce60 con 0x7fdc2006ee30 2024-03-28T15:01:39.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.575+0000 7fd3a8ff9640 1 -- 172.21.15.186:0/642371857 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4018630 con 0x7fd3ac1ed720 2024-03-28T15:01:39.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.575+0000 7fdc1cff9640 1 -- 172.21.15.186:0/4263877904 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc18015070 con 0x7fdc2006ee30 2024-03-28T15:01:39.584 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.575+0000 7fdc1cff9640 1 -- 172.21.15.186:0/4263877904 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc18005070 con 0x7fdc2006ee30 2024-03-28T15:01:39.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.575+0000 7fd3b2447640 1 -- 172.21.15.186:0/642371857 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ac14b600 con 0x7fd3ac1ed720 2024-03-28T15:01:39.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.575+0000 7fd3b2447640 1 -- 172.21.15.186:0/642371857 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ac146ef0 con 0x7fd3ac1ed720 2024-03-28T15:01:39.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.577+0000 7fd3a8ff9640 1 -- 172.21.15.186:0/642371857 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a40187d0 con 0x7fd3ac1ed720 2024-03-28T15:01:39.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.577+0000 7fd3a8ff9640 1 -- 172.21.15.186:0/642371857 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4018ba0 con 0x7fd3ac1ed720 2024-03-28T15:01:39.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.576+0000 7fdc265d6640 1 -- 172.21.15.186:0/4263877904 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc201f3eb0 con 0x7fdc2006ee30 2024-03-28T15:01:39.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.576+0000 7fdc265d6640 1 -- 172.21.15.186:0/4263877904 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc201f4750 con 0x7fdc2006ee30 2024-03-28T15:01:39.586 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.578+0000 7fdc067fc640 1 -- 172.21.15.186:0/4263877904 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc200c23d0 con 0x7fdc2006ee30 2024-03-28T15:01:39.586 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.579+0000 7fd3a8ff9640 1 -- 172.21.15.186:0/642371857 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd3a40213f0 con 0x7fd3ac1ed720 2024-03-28T15:01:39.586 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.579+0000 7fd3b2447640 1 -- 172.21.15.186:0/642371857 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd38c005290 con 0x7fd3ac1ed720 2024-03-28T15:01:39.586 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.580+0000 7fd3a8ff9640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd39806c600 0x7fd39806eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.586 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.580+0000 7fd3abfff640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd39806c600 0x7fd39806eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.580+0000 7fdc1cff9640 1 -- 172.21.15.186:0/4263877904 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdc18005210 con 0x7fdc2006ee30 2024-03-28T15:01:39.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.581+0000 7fdc1cff9640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdc0c06c530 0x7fdc0c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.581+0000 7fdc1cff9640 1 -- 172.21.15.186:0/4263877904 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fdc1808f9b0 con 0x7fdc2006ee30 2024-03-28T15:01:39.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.581+0000 7fdc1f7fe640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdc0c06c530 0x7fdc0c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.588 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.582+0000 7fd3a8ff9640 1 -- 172.21.15.186:0/642371857 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd3a408c0d0 con 0x7fd3ac1ed720 2024-03-28T15:01:39.588 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.582+0000 7fd3abfff640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd39806c600 0x7fd39806eac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009950 tx=0x7fd39c009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.588 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.582+0000 7fdc1f7fe640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdc0c06c530 0x7fdc0c06e9f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdc14036000 tx=0x7fdc14030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.588 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.584+0000 7fdc1cff9640 1 -- 172.21.15.186:0/4263877904 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fdc200c23d0 con 0x7fdc2006ee30 2024-03-28T15:01:39.588 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.585+0000 7fd3a8ff9640 1 -- 172.21.15.186:0/642371857 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd3a4012070 con 0x7fd3ac1ed720 2024-03-28T15:01:39.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.622+0000 7f29ba902640 1 Processor -- start 2024-03-28T15:01:39.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f4d2d93a640 1 Processor -- start 2024-03-28T15:01:39.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.623+0000 7f300ffff640 1 Processor -- start 2024-03-28T15:01:39.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.622+0000 7f29ba902640 1 -- start start 2024-03-28T15:01:39.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f29ba902640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f29ac0e3a40 0x7f29ac0e5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f29ba902640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29ac007310 con 0x7f29ac0dc6a0 2024-03-28T15:01:39.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f29ba902640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29ac0075f0 con 0x7f29ac0d53b0 2024-03-28T15:01:39.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f29ba902640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f29ac0078d0 con 0x7f29ac0e3a40 2024-03-28T15:01:39.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.623+0000 7f300ffff640 1 -- start start 2024-03-28T15:01:39.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f300ffff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f30080d9250 0x7f3008039e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f300ffff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f300804ffd0 0x7f30080dfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f300ffff640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f300803a4b0 con 0x7f30080e0490 2024-03-28T15:01:39.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f300ffff640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f30080dffe0 con 0x7f30080d9250 2024-03-28T15:01:39.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f300ffff640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3008050460 con 0x7f300804ffd0 2024-03-28T15:01:39.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f4d2d93a640 1 -- start start 2024-03-28T15:01:39.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f29b37fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f29ac0dc6a0 0x7f29ac0dcac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51340/0 (socket says 172.21.15.186:51340) 2024-03-28T15:01:39.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.624+0000 7f29b37fe640 1 -- 172.21.15.186:0/4199304726 learned_addr learned my addr 172.21.15.186:0/4199304726 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:39.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f4d2d93a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4d2806f230 0x7f4d280c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f4d2d93a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d280c1e30 0x7f4d2815f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f300f7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f300804ffd0 0x7f30080dfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f300effd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f30080d9250 0x7f3008039e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f300effd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f30080d9250 0x7f3008039e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:37108/0 (socket says 172.21.15.186:37108) 2024-03-28T15:01:39.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f300f7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f300804ffd0 0x7f30080dfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:51616/0 (socket says 172.21.15.186:51616) 2024-03-28T15:01:39.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f300effd640 1 -- 172.21.15.186:0/1320509255 learned_addr learned my addr 172.21.15.186:0/1320509255 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f300e7fc640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f30080e0490 0x7f30080071b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:58542/0 (socket says 172.21.15.186:58542) 2024-03-28T15:01:39.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f4d26ffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4d2806ee50 0x7f4d280c1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51356/0 (socket says 172.21.15.186:51356) 2024-03-28T15:01:39.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 568235412 0 0) 0x7f29ac007310 con 0x7f29ac0dc6a0 2024-03-28T15:01:39.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29a4003520 con 0x7f29ac0dc6a0 2024-03-28T15:01:39.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3433695379 0 0) 0x7f29ac0075f0 con 0x7f29ac0d53b0 2024-03-28T15:01:39.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29ac007310 con 0x7f29ac0d53b0 2024-03-28T15:01:39.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b8e9f640 1 --2- 172.21.15.186:0/4199304726 >> v2:172.21.15.191:3300/0 conn(0x7f29ac0e3a40 0x7f29ac0e5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1121365354 0 0) 0x7f29a4003520 con 0x7f29ac0dc6a0 2024-03-28T15:01:39.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 >> v2:172.21.15.191:3300/0 conn(0x7f29ac0e3a40 msgr2=0x7f29ac0e5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 --2- 172.21.15.186:0/4199304726 >> v2:172.21.15.191:3300/0 conn(0x7f29ac0e3a40 0x7f29ac0e5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 >> v1:172.21.15.191:6789/0 conn(0x7f29ac0d53b0 legacy=0x7f29ac006ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ac007bb0 con 0x7f29ac0dc6a0 2024-03-28T15:01:39.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f29a8001670 con 0x7f29ac0dc6a0 2024-03-28T15:01:39.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f4d26ffd640 1 -- 172.21.15.186:0/432971219 learned_addr learned my addr 172.21.15.186:0/432971219 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:39.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.625+0000 7f4d2d93a640 1 -- 172.21.15.186:0/432971219 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d280c0f40 con 0x7f4d2806ee50 2024-03-28T15:01:39.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f4d2d93a640 1 -- 172.21.15.186:0/432971219 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f4d2815fcf0 con 0x7f4d2806f230 2024-03-28T15:01:39.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f4d2d93a640 1 -- 172.21.15.186:0/432971219 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4d280c1290 con 0x7f4d280c1e30 2024-03-28T15:01:39.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f4d277fe640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d280c1e30 0x7f4d2815f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f4d267fc640 1 --2- 172.21.15.186:0/432971219 >> v2:172.21.15.186:3300/0 conn(0x7f4d2806f230 0x7f4d280c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1364763214 0 0) 0x7f4d280c0f40 con 0x7f4d2806ee50 2024-03-28T15:01:39.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d10003520 con 0x7f4d2806ee50 2024-03-28T15:01:39.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 685165922 0 0) 0x7f4d10003520 con 0x7f4d2806ee50 2024-03-28T15:01:39.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d280c1e30 msgr2=0x7f4d2815f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d280c1e30 0x7f4d2815f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 >> v2:172.21.15.186:3300/0 conn(0x7f4d2806f230 msgr2=0x7f4d280c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 --2- 172.21.15.186:0/432971219 >> v2:172.21.15.186:3300/0 conn(0x7f4d2806f230 0x7f4d280c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d280c7040 con 0x7f4d2806ee50 2024-03-28T15:01:39.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f4d200019f0 con 0x7f4d2806ee50 2024-03-28T15:01:39.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 >> v1:172.21.15.186:6789/0 conn(0x7f4d2806ee50 legacy=0x7f4d280c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f300f7fe640 1 -- 172.21.15.186:0/1320509255 >> v2:172.21.15.191:3300/0 conn(0x7f30080d9250 msgr2=0x7f3008039e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f300f7fe640 1 --2- 172.21.15.186:0/1320509255 >> v2:172.21.15.191:3300/0 conn(0x7f30080d9250 0x7f3008039e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f300f7fe640 1 -- 172.21.15.186:0/1320509255 >> v1:172.21.15.191:6789/0 conn(0x7f30080e0490 legacy=0x7f30080071b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f300f7fe640 1 -- 172.21.15.186:0/1320509255 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300804ef30 con 0x7f300804ffd0 2024-03-28T15:01:39.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 >> v1:172.21.15.186:6789/0 conn(0x7f29ac0dc6a0 legacy=0x7f29ac0dcac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 --2- 172.21.15.186:0/4199304726 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29a4003b90 0x7f29a4003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 --2- 172.21.15.186:0/4199304726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a4004800 0x7f29a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 --2- 172.21.15.186:0/4199304726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a40052e0 0x7f29a4005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac0078d0 con 0x7f29a40052e0 2024-03-28T15:01:39.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f29a4006270 con 0x7f29a4004800 2024-03-28T15:01:39.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f29a4006470 con 0x7f29a4003b90 2024-03-28T15:01:39.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b37fe640 1 --2- 172.21.15.186:0/4199304726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a4004800 0x7f29a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b3fff640 1 --2- 172.21.15.186:0/4199304726 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29a4003b90 0x7f29a4003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b37fe640 1 -- 172.21.15.186:0/4199304726 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29a4003b90 msgr2=0x7f29a4003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b37fe640 1 --2- 172.21.15.186:0/4199304726 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29a4003b90 0x7f29a4003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b37fe640 1 -- 172.21.15.186:0/4199304726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a40052e0 msgr2=0x7f29a4005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b37fe640 1 --2- 172.21.15.186:0/4199304726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a40052e0 0x7f29a4005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.626+0000 7f29b37fe640 1 -- 172.21.15.186:0/4199304726 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a4003520 con 0x7f29a4004800 2024-03-28T15:01:39.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d100039b0 0x7f4d10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d10004800 0x7f4d10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d100051c0 0x7f4d1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2815fcf0 con 0x7f4d10004800 2024-03-28T15:01:39.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4d280c1290 con 0x7f4d100039b0 2024-03-28T15:01:39.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4d1000c010 con 0x7f4d100051c0 2024-03-28T15:01:39.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d267fc640 1 --2- 172.21.15.186:0/432971219 >> v2:172.21.15.186:3300/0 conn(0x7f4d2806f230 0x7f4d280c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:39.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f29b8e9f640 1 -- 172.21.15.186:0/4199304726 reap_dead start 2024-03-28T15:01:39.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d267fc640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d10004800 0x7f4d10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f300f7fe640 1 --2- 172.21.15.186:0/1320509255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f300804ffd0 0x7f30080dfaa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f30040097a0 tx=0x7f300402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=760a120846255148 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f4d277fe640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d100051c0 0x7f4d1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.627+0000 7f29b37fe640 1 --2- 172.21.15.186:0/4199304726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a4004800 0x7f29a4004c80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29a800c0e0 tx=0x7f29a4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8af9ba09f68d81d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.628+0000 7fc14e0df640 1 Processor -- start 2024-03-28T15:01:39.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.628+0000 7f4d267fc640 1 -- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d100051c0 msgr2=0x7f4d1000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.628+0000 7f4d267fc640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d100051c0 0x7f4d1000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.628+0000 7f4d26ffd640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d100039b0 0x7f4d10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.628+0000 7f4d267fc640 1 -- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d100039b0 msgr2=0x7f4d10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.628+0000 7fc14e0df640 1 -- start start 2024-03-28T15:01:39.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.628+0000 7f4d267fc640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d100039b0 0x7f4d10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.628+0000 7f4d267fc640 1 -- 172.21.15.186:0/432971219 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d280c7040 con 0x7f4d10004800 2024-03-28T15:01:39.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7f300dffb640 1 -- 172.21.15.186:0/1320509255 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004035020 con 0x7f300804ffd0 2024-03-28T15:01:39.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7f4d26ffd640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d100039b0 0x7f4d10003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:39.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7f300dffb640 1 -- 172.21.15.186:0/1320509255 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3004031810 con 0x7f300804ffd0 2024-03-28T15:01:39.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7f300dffb640 1 -- 172.21.15.186:0/1320509255 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004031be0 con 0x7f300804ffd0 2024-03-28T15:01:39.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.628+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a8007c00 con 0x7f29a4004800 2024-03-28T15:01:39.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.628+0000 7f29b2ffd640 1 -- 172.21.15.186:0/4199304726 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29a8041070 con 0x7f29a4004800 2024-03-28T15:01:39.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7f29ba902640 1 -- 172.21.15.186:0/4199304726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a4004800 msgr2=0x7f29a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7f29ba902640 1 --2- 172.21.15.186:0/4199304726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a4004800 0x7f29a4004c80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29a800c0e0 tx=0x7f29a4005830 comp rx=0 tx=0).stop 2024-03-28T15:01:39.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7f29ba902640 1 -- 172.21.15.186:0/4199304726 shutdown_connections 2024-03-28T15:01:39.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7f29ba902640 1 --2- 172.21.15.186:0/4199304726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a4004800 0x7f29a4004c80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7f29ba902640 1 -- 172.21.15.186:0/4199304726 >> 172.21.15.186:0/4199304726 conn(0x7f29ac050a10 msgr2=0x7f29ac050e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:39.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7f29ba902640 1 -- 172.21.15.186:0/4199304726 shutdown_connections 2024-03-28T15:01:39.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7f29ba902640 1 -- 172.21.15.186:0/4199304726 wait complete. 2024-03-28T15:01:39.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.630+0000 7f4d267fc640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d10004800 0x7f4d10004c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009c40 tx=0x7f4d10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e12be5f49134ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7fc14e0df640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc14806f230 0x7fc1480c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.630+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18009840 con 0x7f4d10004800 2024-03-28T15:01:39.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.630+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d18004020 con 0x7f4d10004800 2024-03-28T15:01:39.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.630+0000 7f4d25ffb640 1 -- 172.21.15.186:0/432971219 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18004490 con 0x7f4d10004800 2024-03-28T15:01:39.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7fc14e0df640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1480c1e30 0x7fc14815f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7fc14e0df640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1480c0f40 con 0x7fc14806ee50 2024-03-28T15:01:39.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7fc14e0df640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fc14815fcf0 con 0x7fc14806f230 2024-03-28T15:01:39.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7fc14e0df640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc1480c1290 con 0x7fc1480c1e30 2024-03-28T15:01:39.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7fc146ffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc14806f230 0x7fc1480c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7fc146ffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc14806f230 0x7fc1480c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:37142/0 (socket says 172.21.15.186:37142) 2024-03-28T15:01:39.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7fc146ffd640 1 -- 172.21.15.186:0/1340313609 learned_addr learned my addr 172.21.15.186:0/1340313609 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7fc1477fe640 1 --1- 172.21.15.186:0/1340313609 >> v1:172.21.15.191:6789/0 conn(0x7fc14806ee50 0x7fc1480c1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:58548/0 (socket says 172.21.15.186:58548) 2024-03-28T15:01:39.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7fc147fff640 1 --2- 172.21.15.186:0/1340313609 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1480c1e30 0x7fc14815f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.629+0000 7fc1467fc640 1 -- 172.21.15.186:0/1340313609 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2037033817 0 0) 0x7fc1480c0f40 con 0x7fc14806ee50 2024-03-28T15:01:39.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.630+0000 7fc1467fc640 1 -- 172.21.15.186:0/1340313609 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc130014530 con 0x7fc14806ee50 2024-03-28T15:01:39.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.630+0000 7f300ffff640 1 -- 172.21.15.186:0/1320509255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f300804ffd0 msgr2=0x7f30080dfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.630+0000 7f300ffff640 1 --2- 172.21.15.186:0/1320509255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f300804ffd0 0x7f30080dfaa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f30040097a0 tx=0x7f300402b8b0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.630+0000 7fc146ffd640 1 -- 172.21.15.186:0/1340313609 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1480c1e30 msgr2=0x7fc14815f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7f300ffff640 1 -- 172.21.15.186:0/1320509255 shutdown_connections 2024-03-28T15:01:39.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7f300ffff640 1 --2- 172.21.15.186:0/1320509255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f300804ffd0 0x7f30080dfaa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7f300ffff640 1 --2- 172.21.15.186:0/1320509255 >> v2:172.21.15.191:3300/0 conn(0x7f30080d9250 0x7f3008039e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7f300ffff640 1 -- 172.21.15.186:0/1320509255 >> 172.21.15.186:0/1320509255 conn(0x7f300800e160 msgr2=0x7f300800e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:39.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.630+0000 7fc146ffd640 1 --2- 172.21.15.186:0/1340313609 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1480c1e30 0x7fc14815f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.630+0000 7fc146ffd640 1 -- 172.21.15.186:0/1340313609 >> v1:172.21.15.191:6789/0 conn(0x7fc14806ee50 legacy=0x7fc1480c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.630+0000 7fc146ffd640 1 -- 172.21.15.186:0/1340313609 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1480c7040 con 0x7fc14806f230 2024-03-28T15:01:39.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7f29ba902640 1 Processor -- start 2024-03-28T15:01:39.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7fc146ffd640 1 --2- 172.21.15.186:0/1340313609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc14806f230 0x7fc1480c6920 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc140009e50 tx=0x7fc14002c030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ebf1429ea8e15cd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7f4d2d93a640 1 -- 172.21.15.186:0/432971219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d10004800 msgr2=0x7f4d10004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7f4d2d93a640 1 --2- 172.21.15.186:0/432971219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d10004800 0x7f4d10004c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009c40 tx=0x7f4d10005830 comp rx=0 tx=0).stop 2024-03-28T15:01:39.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7f4d277fe640 1 -- 172.21.15.186:0/432971219 reap_dead start 2024-03-28T15:01:39.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7fc1467fc640 1 -- 172.21.15.186:0/1340313609 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14002cbd0 con 0x7fc14806f230 2024-03-28T15:01:39.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7f4d2d93a640 1 -- 172.21.15.186:0/432971219 shutdown_connections 2024-03-28T15:01:39.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7f4d2d93a640 1 -- 172.21.15.186:0/432971219 >> 172.21.15.186:0/432971219 conn(0x7f4d2806c010 msgr2=0x7f4d280cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:39.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7fc1467fc640 1 -- 172.21.15.186:0/1340313609 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc14002cd70 con 0x7fc14806f230 2024-03-28T15:01:39.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7fc1467fc640 1 -- 172.21.15.186:0/1340313609 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1400345f0 con 0x7fc14806f230 2024-03-28T15:01:39.649 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7f29ba902640 1 -- start start 2024-03-28T15:01:39.649 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.631+0000 7fc397623640 1 Processor -- start 2024-03-28T15:01:39.649 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.632+0000 7f300ffff640 1 -- 172.21.15.186:0/1320509255 shutdown_connections 2024-03-28T15:01:39.649 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.632+0000 7f4d2d93a640 1 -- 172.21.15.186:0/432971219 shutdown_connections 2024-03-28T15:01:39.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.632+0000 7f300ffff640 1 -- 172.21.15.186:0/1320509255 wait complete. 2024-03-28T15:01:39.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.632+0000 7fc397623640 1 -- start start 2024-03-28T15:01:39.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.632+0000 7fc14e0df640 1 -- 172.21.15.186:0/1340313609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc14806f230 msgr2=0x7fc1480c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.632+0000 7fc14e0df640 1 --2- 172.21.15.186:0/1340313609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc14806f230 0x7fc1480c6920 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc140009e50 tx=0x7fc14002c030 comp rx=0 tx=0).stop 2024-03-28T15:01:39.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.632+0000 7f4d2d93a640 1 -- 172.21.15.186:0/432971219 wait complete. 2024-03-28T15:01:39.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7f29ba902640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a4004800 0x7f29ac0d2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7f29ba902640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29ac04f750 0x7f29ac0d0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7f29ba902640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29ac0d0cb0 0x7f29ac0c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7fc397623640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc390154a10 0x7fc3900cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7f29ba902640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac04d200 con 0x7f29a4004800 2024-03-28T15:01:39.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7f29ba902640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac008170 con 0x7f29ac0d0cb0 2024-03-28T15:01:39.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7f29ba902640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f29ac0d5790 con 0x7f29ac04f750 2024-03-28T15:01:39.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7fc397623640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc3900cce90 0x7fc3900cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7fc397623640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39015de90 con 0x7fc39015b2c0 2024-03-28T15:01:39.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7fc397623640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc3900c5ae0 con 0x7fc3900cce90 2024-03-28T15:01:39.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7fc397623640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fc39015e750 con 0x7fc390154a10 2024-03-28T15:01:39.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7fc395bc0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc3900cce90 0x7fc3900cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7fc395bc0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc3900cce90 0x7fc3900cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48310/0 (socket says 172.21.15.186:48310) 2024-03-28T15:01:39.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7f29b8e9f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29ac0d0cb0 0x7f29ac0c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7f29b3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a4004800 0x7f29ac0d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7fc395bc0640 1 -- 172.21.15.186:0/3942172254 learned_addr learned my addr 172.21.15.186:0/3942172254 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7f29b3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a4004800 0x7f29ac0d2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48326/0 (socket says 172.21.15.186:48326) 2024-03-28T15:01:39.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7f29b3fff640 1 -- 172.21.15.186:0/419013735 learned_addr learned my addr 172.21.15.186:0/419013735 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7f29b37fe640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29ac04f750 0x7f29ac0d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.633+0000 7fc3953bf640 1 --2- 172.21.15.186:0/3942172254 >> v2:172.21.15.191:3300/0 conn(0x7fc390154a10 0x7fc3900cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7f300ffff640 1 Processor -- start 2024-03-28T15:01:39.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc14e0df640 1 -- 172.21.15.186:0/1340313609 shutdown_connections 2024-03-28T15:01:39.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc14e0df640 1 --2- 172.21.15.186:0/1340313609 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1480c1e30 0x7fc14815f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc14e0df640 1 --2- 172.21.15.186:0/1340313609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc14806f230 0x7fc1480c6920 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc14e0df640 1 -- 172.21.15.186:0/1340313609 >> 172.21.15.186:0/1340313609 conn(0x7fc14806c010 msgr2=0x7fc1480cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:39.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7f5790113640 1 Processor -- start 2024-03-28T15:01:39.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc14e0df640 1 -- 172.21.15.186:0/1340313609 shutdown_connections 2024-03-28T15:01:39.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7f4d2d93a640 1 Processor -- start 2024-03-28T15:01:39.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc387fff640 1 -- 172.21.15.186:0/3942172254 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 29497372 0 0) 0x7fc39015de90 con 0x7fc39015b2c0 2024-03-28T15:01:39.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc387fff640 1 -- 172.21.15.186:0/3942172254 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc37c003520 con 0x7fc39015b2c0 2024-03-28T15:01:39.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7f300ffff640 1 -- start start 2024-03-28T15:01:39.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7f5790113640 1 -- start start 2024-03-28T15:01:39.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7f29b3fff640 1 -- 172.21.15.186:0/419013735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29ac04f750 msgr2=0x7f29ac0d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7f29b3fff640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29ac04f750 0x7f29ac0d0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7f29b3fff640 1 -- 172.21.15.186:0/419013735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29ac0d0cb0 msgr2=0x7f29ac0c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7f29b3fff640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29ac0d0cb0 0x7f29ac0c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc395bc0640 1 -- 172.21.15.186:0/3942172254 >> v2:172.21.15.191:3300/0 conn(0x7fc390154a10 msgr2=0x7fc3900cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc395bc0640 1 --2- 172.21.15.186:0/3942172254 >> v2:172.21.15.191:3300/0 conn(0x7fc390154a10 0x7fc3900cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc387fff640 1 -- 172.21.15.186:0/3942172254 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2279032208 0 0) 0x7fc37c003520 con 0x7fc39015b2c0 2024-03-28T15:01:39.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7f29b3fff640 1 -- 172.21.15.186:0/419013735 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ac0d1130 con 0x7f29a4004800 2024-03-28T15:01:39.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc395bc0640 1 -- 172.21.15.186:0/3942172254 >> v1:172.21.15.191:6789/0 conn(0x7fc39015b2c0 legacy=0x7fc3900c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc395bc0640 1 -- 172.21.15.186:0/3942172254 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3900cb6d0 con 0x7fc3900cce90 2024-03-28T15:01:39.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f29b3fff640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a4004800 0x7f29ac0d2320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f29a000e810 tx=0x7f29a000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f29b0ff9640 1 -- 172.21.15.186:0/419013735 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a00050d0 con 0x7f29a4004800 2024-03-28T15:01:39.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f29ba902640 1 -- 172.21.15.186:0/419013735 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ac04a9c0 con 0x7f29a4004800 2024-03-28T15:01:39.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f29ba902640 1 -- 172.21.15.186:0/419013735 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ac048960 con 0x7f29a4004800 2024-03-28T15:01:39.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f300ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f300804ffd0 0x7f3008175b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f300ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30080d9250 0x7f3008168b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f300ffff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30080e0490 0x7f30080d0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f300ffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f300804f7d0 con 0x7f30080d9250 2024-03-28T15:01:39.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f300ffff640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f300804f4f0 con 0x7f300804ffd0 2024-03-28T15:01:39.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f300ffff640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f300804fab0 con 0x7f30080e0490 2024-03-28T15:01:39.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f5790113640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5788155470 0x7f578815ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f5790113640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f57880b39c0 0x7f57880ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f5790113640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f578815fb00 con 0x7f5788155850 2024-03-28T15:01:39.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f5790113640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f57880b3530 con 0x7f57880b39c0 2024-03-28T15:01:39.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f5790113640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f578815f260 con 0x7f5788155470 2024-03-28T15:01:39.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7f4d2d93a640 1 -- start start 2024-03-28T15:01:39.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f4d2d93a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d2814f6d0 0x7f4d2815cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f4d2d93a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d280c5990 0x7f4d280c35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f300f7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30080e0490 0x7f30080d0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f300f7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30080e0490 0x7f30080d0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:51680/0 (socket says 172.21.15.186:51680) 2024-03-28T15:01:39.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f300f7fe640 1 -- 172.21.15.186:0/1270582796 learned_addr learned my addr 172.21.15.186:0/1270582796 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f578d6ae640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f5788155850 0x7f57880cc3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:58568/0 (socket says 172.21.15.186:58568) 2024-03-28T15:01:39.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f578d6ae640 1 -- 172.21.15.186:0/794361423 learned_addr learned my addr 172.21.15.186:0/794361423 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:39.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f578deaf640 1 --2- 172.21.15.186:0/794361423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5788155470 0x7f578815ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f300effd640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f300804ffd0 0x7f3008175b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f4d26ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d2814f6d0 0x7f4d2815cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f578cead640 1 -- 172.21.15.186:0/794361423 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3774003652 0 0) 0x7f5780001490 con 0x7f5788155850 2024-03-28T15:01:39.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f578cead640 1 -- 172.21.15.186:0/794361423 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5774003520 con 0x7f5788155850 2024-03-28T15:01:39.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f578deaf640 1 -- 172.21.15.186:0/794361423 >> v2:172.21.15.186:3300/0 conn(0x7f57880b39c0 msgr2=0x7f57880ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:39.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f4d26ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d2814f6d0 0x7f4d2815cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48370/0 (socket says 172.21.15.186:48370) 2024-03-28T15:01:39.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.634+0000 7fc14e0df640 1 -- 172.21.15.186:0/1340313609 wait complete. 2024-03-28T15:01:39.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f300e7fc640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30080d9250 0x7f3008168b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7fc14e0df640 1 Processor -- start 2024-03-28T15:01:39.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7fc395bc0640 1 --2- 172.21.15.186:0/3942172254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3900cce90 0x7fc3900cd2c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc3900d1130 tx=0x7fc38802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d117598f6539fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f29b0ff9640 1 -- 172.21.15.186:0/419013735 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29a0018070 con 0x7f29a4004800 2024-03-28T15:01:39.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f4d2d93a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d2815e570 0x7f4d281f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.635+0000 7f29b0ff9640 1 -- 172.21.15.186:0/419013735 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a000c660 con 0x7f29a4004800 2024-03-28T15:01:39.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f4d267fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d280c5990 0x7f4d280c35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.667 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f4d267fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d280c5990 0x7f4d280c35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:51696/0 (socket says 172.21.15.186:51696) 2024-03-28T15:01:39.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f4d2d93a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d280c78e0 con 0x7f4d2814f6d0 2024-03-28T15:01:39.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f4d2d93a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4d280c7600 con 0x7f4d2815e570 2024-03-28T15:01:39.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f4d2d93a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2806e2b0 con 0x7f4d280c5990 2024-03-28T15:01:39.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f300f7fe640 1 -- 172.21.15.186:0/1270582796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f300804ffd0 msgr2=0x7f3008175b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f300f7fe640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f300804ffd0 0x7f3008175b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f300f7fe640 1 -- 172.21.15.186:0/1270582796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30080d9250 msgr2=0x7f3008168b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f300f7fe640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30080d9250 0x7f3008168b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f300f7fe640 1 -- 172.21.15.186:0/1270582796 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30080d2ad0 con 0x7f30080e0490 2024-03-28T15:01:39.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d2815e570 0x7f4d281f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d2815e570 0x7f4d281f4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:37190/0 (socket says 172.21.15.186:37190) 2024-03-28T15:01:39.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f4d277fe640 1 -- 172.21.15.186:0/1359836887 learned_addr learned my addr 172.21.15.186:0/1359836887 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f300f7fe640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30080e0490 0x7f30080d0e30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f30040098c0 tx=0x7f30040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f578e6b0640 1 --2- 172.21.15.186:0/794361423 >> v2:172.21.15.186:3300/0 conn(0x7f57880b39c0 0x7f57880ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f578deaf640 1 --2- 172.21.15.186:0/794361423 >> v2:172.21.15.186:3300/0 conn(0x7f57880b39c0 0x7f57880ccdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f578deaf640 1 -- 172.21.15.186:0/794361423 >> v1:172.21.15.191:6789/0 conn(0x7f5788155850 legacy=0x7f57880cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7f578deaf640 1 -- 172.21.15.186:0/794361423 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578815f540 con 0x7f5788155470 2024-03-28T15:01:39.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f578e6b0640 1 --2- 172.21.15.186:0/794361423 >> v2:172.21.15.186:3300/0 conn(0x7f57880b39c0 0x7f57880ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:39.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f2ff7fff640 1 -- 172.21.15.186:0/1270582796 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004041070 con 0x7f30080e0490 2024-03-28T15:01:39.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.636+0000 7fc14e0df640 1 -- start start 2024-03-28T15:01:39.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7fc387fff640 1 -- 172.21.15.186:0/3942172254 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380001650 con 0x7fc3900cce90 2024-03-28T15:01:39.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f300ffff640 1 -- 172.21.15.186:0/1270582796 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3008005b90 con 0x7f30080e0490 2024-03-28T15:01:39.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f300ffff640 1 -- 172.21.15.186:0/1270582796 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3008006150 con 0x7f30080e0490 2024-03-28T15:01:39.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f578deaf640 1 --2- 172.21.15.186:0/794361423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5788155470 0x7f578815ebb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f57840097a0 tx=0x7f578402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b018a94785d5ff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f2ff7fff640 1 -- 172.21.15.186:0/1270582796 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f300403ac00 con 0x7f30080e0490 2024-03-28T15:01:39.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f4d267fc640 1 -- 172.21.15.186:0/1359836887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d2815e570 msgr2=0x7f4d281f4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f4d267fc640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d2815e570 0x7f4d281f4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f4d267fc640 1 -- 172.21.15.186:0/1359836887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d2814f6d0 msgr2=0x7f4d2815cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f4d267fc640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d2814f6d0 0x7f4d2815cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f4d267fc640 1 -- 172.21.15.186:0/1359836887 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d280736c0 con 0x7f4d280c5990 2024-03-28T15:01:39.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7fc14e0df640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc14806ee50 0x7fc14814eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc14e0df640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc14806f230 0x7fc1480722e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7f4d267fc640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d280c5990 0x7f4d280c35f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4d18033000 tx=0x7f4d18004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.637+0000 7fc387fff640 1 -- 172.21.15.186:0/3942172254 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc38802c7f0 con 0x7fc3900cce90 2024-03-28T15:01:39.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc14e0df640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1480c1e30 0x7fc148072cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7f578cead640 1 -- 172.21.15.186:0/794361423 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784035020 con 0x7f5788155470 2024-03-28T15:01:39.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7f578cead640 1 -- 172.21.15.186:0/794361423 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5784031830 con 0x7f5788155470 2024-03-28T15:01:39.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7f578cead640 1 -- 172.21.15.186:0/794361423 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784031c40 con 0x7f5788155470 2024-03-28T15:01:39.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc146ffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc14806f230 0x7fc1480722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc1477fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc14806ee50 0x7fc14814eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc147fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1480c1e30 0x7fc148072cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc1477fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc14806ee50 0x7fc14814eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:37202/0 (socket says 172.21.15.186:37202) 2024-03-28T15:01:39.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc147fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1480c1e30 0x7fc148072cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48384/0 (socket says 172.21.15.186:48384) 2024-03-28T15:01:39.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc1477fe640 1 -- 172.21.15.186:0/175225074 learned_addr learned my addr 172.21.15.186:0/175225074 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7f2ff7fff640 1 -- 172.21.15.186:0/1270582796 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004035040 con 0x7f30080e0490 2024-03-28T15:01:39.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc397623640 1 -- 172.21.15.186:0/3942172254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3900cce90 msgr2=0x7fc3900cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc387fff640 1 -- 172.21.15.186:0/3942172254 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380001650 con 0x7fc3900cce90 2024-03-28T15:01:39.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc397623640 1 --2- 172.21.15.186:0/3942172254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3900cce90 0x7fc3900cd2c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc3900d1130 tx=0x7fc38802bed0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f5790113640 1 -- 172.21.15.186:0/794361423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5788155470 msgr2=0x7f578815ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7f4d0ffff640 1 -- 172.21.15.186:0/1359836887 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1802c950 con 0x7f4d280c5990 2024-03-28T15:01:39.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7f4d0ffff640 1 -- 172.21.15.186:0/1359836887 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d1802caf0 con 0x7f4d280c5990 2024-03-28T15:01:39.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7f4d0ffff640 1 -- 172.21.15.186:0/1359836887 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18003610 con 0x7f4d280c5990 2024-03-28T15:01:39.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f4d2d93a640 1 -- 172.21.15.186:0/1359836887 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d280739a0 con 0x7f4d280c5990 2024-03-28T15:01:39.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f4d2d93a640 1 -- 172.21.15.186:0/1359836887 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d2815ef20 con 0x7f4d280c5990 2024-03-28T15:01:39.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f300ffff640 1 -- 172.21.15.186:0/1270582796 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ff0005290 con 0x7f30080e0490 2024-03-28T15:01:39.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc397623640 1 -- 172.21.15.186:0/3942172254 shutdown_connections 2024-03-28T15:01:39.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc397623640 1 --2- 172.21.15.186:0/3942172254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3900cce90 0x7fc3900cd2c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc397623640 1 --2- 172.21.15.186:0/3942172254 >> v2:172.21.15.191:3300/0 conn(0x7fc390154a10 0x7fc3900cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc397623640 1 -- 172.21.15.186:0/3942172254 >> 172.21.15.186:0/3942172254 conn(0x7fc39006b640 msgr2=0x7fc39006ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:39.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f5790113640 1 --2- 172.21.15.186:0/794361423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5788155470 0x7f578815ebb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f57840097a0 tx=0x7f578402b8d0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f5790113640 1 -- 172.21.15.186:0/794361423 shutdown_connections 2024-03-28T15:01:39.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f5790113640 1 --2- 172.21.15.186:0/794361423 >> v2:172.21.15.186:3300/0 conn(0x7f57880b39c0 0x7f57880ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f5790113640 1 --2- 172.21.15.186:0/794361423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5788155470 0x7f578815ebb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc397623640 1 -- 172.21.15.186:0/3942172254 shutdown_connections 2024-03-28T15:01:39.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f5790113640 1 -- 172.21.15.186:0/794361423 >> 172.21.15.186:0/794361423 conn(0x7f578806b640 msgr2=0x7f578806ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:39.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f5790113640 1 -- 172.21.15.186:0/794361423 shutdown_connections 2024-03-28T15:01:39.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f5790113640 1 -- 172.21.15.186:0/794361423 wait complete. 2024-03-28T15:01:39.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc1480c78e0 con 0x7fc1480c1e30 2024-03-28T15:01:39.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc1480c7600 con 0x7fc14806ee50 2024-03-28T15:01:39.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.638+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc14806e2b0 con 0x7fc14806f230 2024-03-28T15:01:39.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc147fff640 1 -- 172.21.15.186:0/175225074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc14806f230 msgr2=0x7fc1480722e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc147fff640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc14806f230 0x7fc1480722e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc147fff640 1 -- 172.21.15.186:0/175225074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc14806ee50 msgr2=0x7fc14814eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc147fff640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc14806ee50 0x7fc14814eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc147fff640 1 -- 172.21.15.186:0/175225074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc148153260 con 0x7fc1480c1e30 2024-03-28T15:01:39.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc147fff640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1480c1e30 0x7fc148072cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc13800d800 tx=0x7fc13800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7f2ff7fff640 1 -- 172.21.15.186:0/1270582796 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3004038020 con 0x7f30080e0490 2024-03-28T15:01:39.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.640+0000 7f29ba902640 1 -- 172.21.15.186:0/419013735 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2994005290 con 0x7f29a4004800 2024-03-28T15:01:39.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.640+0000 7f29b0ff9640 1 -- 172.21.15.186:0/419013735 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f29a000c800 con 0x7f29a4004800 2024-03-28T15:01:39.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.640+0000 7f2ff7fff640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2ffc06c630 0x7f2ffc06eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.639+0000 7fc397623640 1 -- 172.21.15.186:0/3942172254 wait complete. 2024-03-28T15:01:39.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.640+0000 7f4d0dffb640 1 -- 172.21.15.186:0/1359836887 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d00005290 con 0x7f4d280c5990 2024-03-28T15:01:39.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.640+0000 7f29b0ff9640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f299806c520 0x7f299806e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7f29b0ff9640 1 -- 172.21.15.186:0/419013735 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f29a008b2d0 con 0x7f29a4004800 2024-03-28T15:01:39.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7f29b37fe640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f299806c520 0x7f299806e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7f5790113640 1 Processor -- start 2024-03-28T15:01:39.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7f5790113640 1 -- start start 2024-03-28T15:01:39.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7f2ff7fff640 1 -- 172.21.15.186:0/1270582796 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f30040ad1e0 con 0x7f30080e0490 2024-03-28T15:01:39.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7f300effd640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2ffc06c630 0x7f2ffc06eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7fc12ffff640 1 -- 172.21.15.186:0/175225074 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138004020 con 0x7fc1480c1e30 2024-03-28T15:01:39.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7fc12ffff640 1 -- 172.21.15.186:0/175225074 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1380041c0 con 0x7fc1480c1e30 2024-03-28T15:01:39.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7fc12ffff640 1 -- 172.21.15.186:0/175225074 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138004390 con 0x7fc1480c1e30 2024-03-28T15:01:39.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1480cbc90 con 0x7fc1480c1e30 2024-03-28T15:01:39.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc148073980 con 0x7fc1480c1e30 2024-03-28T15:01:39.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7f29b37fe640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f299806c520 0x7f299806e9e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f29a8045000 tx=0x7f29a8009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7fc397623640 1 Processor -- start 2024-03-28T15:01:39.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7f5790113640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57880b39c0 0x7f578815dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7f300effd640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2ffc06c630 0x7f2ffc06eaf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3000009950 tx=0x7f3000008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7f5790113640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5788155470 0x7f5788161860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7fc397623640 1 -- start start 2024-03-28T15:01:39.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7f5790113640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5788161da0 0x7f578815ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7f5790113640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57880c51e0 con 0x7f5788155470 2024-03-28T15:01:39.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7f5790113640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f57880b31c0 con 0x7f5788161da0 2024-03-28T15:01:39.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7f5790113640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f57880c5360 con 0x7f57880b39c0 2024-03-28T15:01:39.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.641+0000 7f4d0ffff640 1 -- 172.21.15.186:0/1359836887 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4d1803d040 con 0x7f4d280c5990 2024-03-28T15:01:39.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7f4d0ffff640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4d1406c600 0x7f4d1406eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7f4d0ffff640 1 -- 172.21.15.186:0/1359836887 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4d180ad1a0 con 0x7f4d280c5990 2024-03-28T15:01:39.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.642+0000 7f4d26ffd640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4d1406c600 0x7f4d1406eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578deaf640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57880b39c0 0x7f578815dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578deaf640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57880b39c0 0x7f578815dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:51714/0 (socket says 172.21.15.186:51714) 2024-03-28T15:01:39.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578deaf640 1 -- 172.21.15.186:0/4107138708 learned_addr learned my addr 172.21.15.186:0/4107138708 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f4d26ffd640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4d1406c600 0x7f4d1406eac0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4d20001ba0 tx=0x7f4d20001cf0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578e6b0640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5788161da0 0x7f578815ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578d6ae640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5788155470 0x7f5788161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578deaf640 1 -- 172.21.15.186:0/4107138708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5788161da0 msgr2=0x7f578815ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578deaf640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5788161da0 0x7f578815ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578deaf640 1 -- 172.21.15.186:0/4107138708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5788155470 msgr2=0x7f5788161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578deaf640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5788155470 0x7f5788161860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578deaf640 1 -- 172.21.15.186:0/4107138708 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578815e970 con 0x7f57880b39c0 2024-03-28T15:01:39.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578d6ae640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5788155470 0x7f5788161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:39.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7f578deaf640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57880b39c0 0x7f578815dce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5784009890 tx=0x7f57840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7fc397623640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3900cce90 0x7fc39014b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7fc397623640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390154a10 0x7fc3900ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7fc397623640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc39015b2c0 0x7fc390159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7fc397623640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc3900cb9b0 con 0x7fc390154a10 2024-03-28T15:01:39.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7fc397623640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc39015e3e0 con 0x7fc3900cce90 2024-03-28T15:01:39.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7fc397623640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc39015b710 con 0x7fc39015b2c0 2024-03-28T15:01:39.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7fc3953bf640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3900cce90 0x7fc39014b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7fc394bbe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390154a10 0x7fc3900ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7fc3953bf640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3900cce90 0x7fc39014b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:37220/0 (socket says 172.21.15.186:37220) 2024-03-28T15:01:39.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7fc3953bf640 1 -- 172.21.15.186:0/138999098 learned_addr learned my addr 172.21.15.186:0/138999098 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:39.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.643+0000 7fc394bbe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390154a10 0x7fc3900ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48388/0 (socket says 172.21.15.186:48388) 2024-03-28T15:01:39.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.644+0000 7fc395bc0640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc39015b2c0 0x7fc390159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.644+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc14806e680 con 0x7fc1480c1e30 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.644+0000 7fc3953bf640 1 -- 172.21.15.186:0/138999098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc39015b2c0 msgr2=0x7fc390159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.644+0000 7fc3953bf640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc39015b2c0 0x7fc390159d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.644+0000 7fc3953bf640 1 -- 172.21.15.186:0/138999098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390154a10 msgr2=0x7fc3900ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.644+0000 7fc3953bf640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390154a10 0x7fc3900ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.644+0000 7fc3953bf640 1 -- 172.21.15.186:0/138999098 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3900d0c50 con 0x7fc3900cce90 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.644+0000 7fc394bbe640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390154a10 0x7fc3900ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.644+0000 7fc395bc0640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc39015b2c0 0x7fc390159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.644+0000 7fc3953bf640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3900cce90 0x7fc39014b6f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00e9b0 tx=0x7fc38c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.644+0000 7fc385ffb640 1 -- 172.21.15.186:0/138999098 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c017070 con 0x7fc3900cce90 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.645+0000 7fc385ffb640 1 -- 172.21.15.186:0/138999098 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc38c00c9a0 con 0x7fc3900cce90 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.645+0000 7fc385ffb640 1 -- 172.21.15.186:0/138999098 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c00cd90 con 0x7fc3900cce90 2024-03-28T15:01:39.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.645+0000 7fc397623640 1 -- 172.21.15.186:0/138999098 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3900d0f30 con 0x7fc3900cce90 2024-03-28T15:01:39.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.645+0000 7f2ff7fff640 1 -- 172.21.15.186:0/1270582796 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f300407b220 con 0x7f30080e0490 2024-03-28T15:01:39.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.645+0000 7fc397623640 1 -- 172.21.15.186:0/138999098 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc39014e210 con 0x7fc3900cce90 2024-03-28T15:01:39.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.646+0000 7fc397623640 1 -- 172.21.15.186:0/138999098 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3900b3280 con 0x7fc3900cce90 2024-03-28T15:01:39.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.647+0000 7fc385ffb640 1 -- 172.21.15.186:0/138999098 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc38c01a070 con 0x7fc3900cce90 2024-03-28T15:01:39.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.647+0000 7fc385ffb640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc37406c560 0x7fc37406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.647+0000 7fc385ffb640 1 -- 172.21.15.186:0/138999098 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc38c08b980 con 0x7fc3900cce90 2024-03-28T15:01:39.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.648+0000 7fc394bbe640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc37406c560 0x7fc37406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.648+0000 7fc394bbe640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc37406c560 0x7fc37406ea20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc3800017c0 tx=0x7fc380009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.648+0000 7f29b0ff9640 1 -- 172.21.15.186:0/419013735 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f29ac0d1130 con 0x7f29a4004800 2024-03-28T15:01:39.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.649+0000 7f577e7fc640 1 -- 172.21.15.186:0/4107138708 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784048070 con 0x7f57880b39c0 2024-03-28T15:01:39.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.649+0000 7f577e7fc640 1 -- 172.21.15.186:0/4107138708 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f578403a6f0 con 0x7f57880b39c0 2024-03-28T15:01:39.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.649+0000 7f577e7fc640 1 -- 172.21.15.186:0/4107138708 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578403aae0 con 0x7f57880b39c0 2024-03-28T15:01:39.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.650+0000 7f5790113640 1 -- 172.21.15.186:0/4107138708 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57881f4510 con 0x7f57880b39c0 2024-03-28T15:01:39.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.650+0000 7f5790113640 1 -- 172.21.15.186:0/4107138708 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f57881f4ad0 con 0x7f57880b39c0 2024-03-28T15:01:39.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.650+0000 7fc12ffff640 1 -- 172.21.15.186:0/175225074 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc13801b020 con 0x7fc1480c1e30 2024-03-28T15:01:39.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.651+0000 7fc12ffff640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc13406c600 0x7fc13406eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.651+0000 7fc12ffff640 1 -- 172.21.15.186:0/175225074 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc13808bdb0 con 0x7fc1480c1e30 2024-03-28T15:01:39.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.649+0000 7f4d0ffff640 1 -- 172.21.15.186:0/1359836887 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4d1807b150 con 0x7f4d280c5990 2024-03-28T15:01:39.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.651+0000 7fc1477fe640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc13406c600 0x7fc13406eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.652+0000 7fc1477fe640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc13406c600 0x7fc13406eac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc13c007fd0 tx=0x7fc13c007d80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.652+0000 7f5790113640 1 -- 172.21.15.186:0/4107138708 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5788069570 con 0x7f57880b39c0 2024-03-28T15:01:39.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.653+0000 7fc385ffb640 1 -- 172.21.15.186:0/138999098 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc3900b3280 con 0x7fc3900cce90 2024-03-28T15:01:39.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.653+0000 7f577e7fc640 1 -- 172.21.15.186:0/4107138708 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f578404b070 con 0x7f57880b39c0 2024-03-28T15:01:39.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.653+0000 7f577e7fc640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f576c06c600 0x7f576c06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:39.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.654+0000 7f577e7fc640 1 -- 172.21.15.186:0/4107138708 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f57840ace70 con 0x7f57880b39c0 2024-03-28T15:01:39.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.654+0000 7f578d6ae640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f576c06c600 0x7f576c06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:39.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.655+0000 7f578d6ae640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f576c06c600 0x7f576c06eac0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5788154740 tx=0x7f5780009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:39.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.657+0000 7fc12ffff640 1 -- 172.21.15.186:0/175225074 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc14806e680 con 0x7fc1480c1e30 2024-03-28T15:01:39.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.658+0000 7f577e7fc640 1 -- 172.21.15.186:0/4107138708 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5784031800 con 0x7f57880b39c0 2024-03-28T15:01:39.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.995+0000 7fdc067fc640 1 -- 172.21.15.186:0/4263877904 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fdc2006e2d0 con 0x7fdc2006ee30 2024-03-28T15:01:39.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.996+0000 7fdc1cff9640 1 -- 172.21.15.186:0/4263877904 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdc18029080 con 0x7fdc2006ee30 2024-03-28T15:01:39.996 INFO:teuthology.orchestra.run.smithi186.stdout:55834574856 2024-03-28T15:01:40.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.999+0000 7fdc265d6640 1 -- 172.21.15.186:0/4263877904 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdc0c06c530 msgr2=0x7fdc0c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.999+0000 7fdc265d6640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdc0c06c530 0x7fdc0c06e9f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdc14036000 tx=0x7fdc14030000 comp rx=0 tx=0).stop 2024-03-28T15:01:40.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:39.999+0000 7fdc265d6640 1 -- 172.21.15.186:0/4263877904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc2006ee30 msgr2=0x7fdc201667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.000+0000 7fdc265d6640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc2006ee30 0x7fdc201667e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800c750 tx=0x7fdc1800cc10 comp rx=0 tx=0).stop 2024-03-28T15:01:40.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.000+0000 7fdc265d6640 1 -- 172.21.15.186:0/4263877904 shutdown_connections 2024-03-28T15:01:40.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.000+0000 7fdc265d6640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdc0c06c530 0x7fdc0c06e9f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.000+0000 7fdc265d6640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc20169e00 0x7fdc20200fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.000+0000 7fdc265d6640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc2006f210 0x7fdc201fddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.000+0000 7fdc265d6640 1 --2- 172.21.15.186:0/4263877904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc2006ee30 0x7fdc201667e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.000+0000 7fdc265d6640 1 -- 172.21.15.186:0/4263877904 >> 172.21.15.186:0/4263877904 conn(0x7fdc2006c010 msgr2=0x7fdc20163250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.000+0000 7fdc265d6640 1 -- 172.21.15.186:0/4263877904 shutdown_connections 2024-03-28T15:01:40.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.000+0000 7fdc265d6640 1 -- 172.21.15.186:0/4263877904 wait complete. 2024-03-28T15:01:40.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.001+0000 7fd3b2447640 1 -- 172.21.15.186:0/642371857 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd38c005580 con 0x7fd3ac1ed720 2024-03-28T15:01:40.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.001+0000 7fd3a8ff9640 1 -- 172.21.15.186:0/642371857 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd3a405a7a0 con 0x7fd3ac1ed720 2024-03-28T15:01:40.002 INFO:teuthology.orchestra.run.smithi186.stdout:55834574856 2024-03-28T15:01:40.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.005+0000 7fd3927fc640 1 -- 172.21.15.186:0/642371857 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd39806c600 msgr2=0x7fd39806eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.005+0000 7fd3927fc640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd39806c600 0x7fd39806eac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009950 tx=0x7fd39c009060 comp rx=0 tx=0).stop 2024-03-28T15:01:40.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.005+0000 7fd3927fc640 1 -- 172.21.15.186:0/642371857 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3ac1ed720 msgr2=0x7fd3ac1efe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.005+0000 7fd3927fc640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3ac1ed720 0x7fd3ac1efe70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4004710 tx=0x7fd3a400a6a0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.005+0000 7fd3927fc640 1 -- 172.21.15.186:0/642371857 shutdown_connections 2024-03-28T15:01:40.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.006+0000 7fd3927fc640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd39806c600 0x7fd39806eac0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.006+0000 7fd3927fc640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3ac1f03b0 0x7fd3ac145990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.006+0000 7fd3927fc640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3ac1ed720 0x7fd3ac1efe70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.006+0000 7fd3927fc640 1 --2- 172.21.15.186:0/642371857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac1efa90 0x7fd3ac148710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.006+0000 7fd3927fc640 1 -- 172.21.15.186:0/642371857 >> 172.21.15.186:0/642371857 conn(0x7fd3ac06c010 msgr2=0x7fd3ac0cb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.006+0000 7fd3927fc640 1 -- 172.21.15.186:0/642371857 shutdown_connections 2024-03-28T15:01:40.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.006+0000 7fd3927fc640 1 -- 172.21.15.186:0/642371857 wait complete. 2024-03-28T15:01:40.020 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2024-03-28T15:01:40.020 DEBUG:teuthology.parallel:result is None 2024-03-28T15:01:40.023 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2024-03-28T15:01:40.024 DEBUG:teuthology.parallel:result is None 2024-03-28T15:01:40.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.039+0000 7f4d0dffb640 1 -- 172.21.15.186:0/1359836887 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f4d00005580 con 0x7f4d280c5990 2024-03-28T15:01:40.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.040+0000 7f29ba902640 1 -- 172.21.15.186:0/419013735 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f2994005850 con 0x7f29a4004800 2024-03-28T15:01:40.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.040+0000 7f4d0ffff640 1 -- 172.21.15.186:0/1359836887 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4d1807f450 con 0x7f4d280c5990 2024-03-28T15:01:40.040 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2024-03-28T15:01:40.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.040+0000 7f29b0ff9640 1 -- 172.21.15.186:0/419013735 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f29a00599d0 con 0x7f29a4004800 2024-03-28T15:01:40.041 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2024-03-28T15:01:40.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f4d0dffb640 1 -- 172.21.15.186:0/1359836887 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4d1406c600 msgr2=0x7f4d1406eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f4d0dffb640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4d1406c600 0x7f4d1406eac0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4d20001ba0 tx=0x7f4d20001cf0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f4d0dffb640 1 -- 172.21.15.186:0/1359836887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d280c5990 msgr2=0x7f4d280c35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f4d0dffb640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d280c5990 0x7f4d280c35f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4d18033000 tx=0x7f4d18004730 comp rx=0 tx=0).stop 2024-03-28T15:01:40.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f4d0dffb640 1 -- 172.21.15.186:0/1359836887 shutdown_connections 2024-03-28T15:01:40.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f4d0dffb640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4d1406c600 0x7f4d1406eac0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f4d0dffb640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d2815e570 0x7f4d281f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f4d0dffb640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d280c5990 0x7f4d280c35f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f4d0dffb640 1 --2- 172.21.15.186:0/1359836887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d2814f6d0 0x7f4d2815cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f4d0dffb640 1 -- 172.21.15.186:0/1359836887 >> 172.21.15.186:0/1359836887 conn(0x7f4d2806c010 msgr2=0x7f4d280cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f29ba902640 1 -- 172.21.15.186:0/419013735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f299806c520 msgr2=0x7f299806e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f29ba902640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f299806c520 0x7f299806e9e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f29a8045000 tx=0x7f29a8009030 comp rx=0 tx=0).stop 2024-03-28T15:01:40.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f29ba902640 1 -- 172.21.15.186:0/419013735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a4004800 msgr2=0x7f29ac0d2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.042+0000 7f29ba902640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a4004800 0x7f29ac0d2320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f29a000e810 tx=0x7f29a000ecd0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.043+0000 7f29ba902640 1 -- 172.21.15.186:0/419013735 shutdown_connections 2024-03-28T15:01:40.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.043+0000 7f29ba902640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f299806c520 0x7f299806e9e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.043+0000 7f29ba902640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29ac0d0cb0 0x7f29ac0c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.043+0000 7f29ba902640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29ac04f750 0x7f29ac0d0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.043+0000 7f29ba902640 1 --2- 172.21.15.186:0/419013735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a4004800 0x7f29ac0d2320 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.043+0000 7f29ba902640 1 -- 172.21.15.186:0/419013735 >> 172.21.15.186:0/419013735 conn(0x7f29ac050a10 msgr2=0x7f29ac0d2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.043+0000 7f4d0dffb640 1 -- 172.21.15.186:0/1359836887 shutdown_connections 2024-03-28T15:01:40.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.043+0000 7f4d0dffb640 1 -- 172.21.15.186:0/1359836887 wait complete. 2024-03-28T15:01:40.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.043+0000 7f29ba902640 1 -- 172.21.15.186:0/419013735 shutdown_connections 2024-03-28T15:01:40.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.043+0000 7f29ba902640 1 -- 172.21.15.186:0/419013735 wait complete. 2024-03-28T15:01:40.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.054+0000 7f300ffff640 1 -- 172.21.15.186:0/1270582796 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f2ff0005580 con 0x7f30080e0490 2024-03-28T15:01:40.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.054+0000 7f2ff7fff640 1 -- 172.21.15.186:0/1270582796 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f300407f520 con 0x7f30080e0490 2024-03-28T15:01:40.055 INFO:teuthology.orchestra.run.smithi186.stdout:55834574856 2024-03-28T15:01:40.055 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2024-03-28T15:01:40.056 DEBUG:teuthology.parallel:result is None 2024-03-28T15:01:40.056 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2024-03-28T15:01:40.056 DEBUG:teuthology.parallel:result is None 2024-03-28T15:01:40.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.057+0000 7f300ffff640 1 -- 172.21.15.186:0/1270582796 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2ffc06c630 msgr2=0x7f2ffc06eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.058+0000 7f300ffff640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2ffc06c630 0x7f2ffc06eaf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3000009950 tx=0x7f3000008040 comp rx=0 tx=0).stop 2024-03-28T15:01:40.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.058+0000 7f300ffff640 1 -- 172.21.15.186:0/1270582796 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30080e0490 msgr2=0x7f30080d0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.058+0000 7f300ffff640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30080e0490 0x7f30080d0e30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f30040098c0 tx=0x7f30040046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.058+0000 7f300ffff640 1 -- 172.21.15.186:0/1270582796 shutdown_connections 2024-03-28T15:01:40.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.058+0000 7f300ffff640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f2ffc06c630 0x7f2ffc06eaf0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.058+0000 7f300ffff640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30080e0490 0x7f30080d0e30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.058+0000 7f300ffff640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30080d9250 0x7f3008168b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.058+0000 7f300ffff640 1 --2- 172.21.15.186:0/1270582796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f300804ffd0 0x7f3008175b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.058+0000 7f300ffff640 1 -- 172.21.15.186:0/1270582796 >> 172.21.15.186:0/1270582796 conn(0x7f300800e160 msgr2=0x7f3008053880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.058+0000 7f300ffff640 1 -- 172.21.15.186:0/1270582796 shutdown_connections 2024-03-28T15:01:40.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.058+0000 7f300ffff640 1 -- 172.21.15.186:0/1270582796 wait complete. 2024-03-28T15:01:40.073 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2024-03-28T15:01:40.073 DEBUG:teuthology.parallel:result is None 2024-03-28T15:01:40.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.075+0000 7f5790113640 1 -- 172.21.15.186:0/4107138708 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f578806bcc0 con 0x7f57880b39c0 2024-03-28T15:01:40.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.076+0000 7f577e7fc640 1 -- 172.21.15.186:0/4107138708 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f578407b500 con 0x7f57880b39c0 2024-03-28T15:01:40.076 INFO:teuthology.orchestra.run.smithi186.stdout:55834574857 2024-03-28T15:01:40.077 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.077+0000 7fc397623640 1 -- 172.21.15.186:0/138999098 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc390069570 con 0x7fc3900cce90 2024-03-28T15:01:40.078 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.077+0000 7fc385ffb640 1 -- 172.21.15.186:0/138999098 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc3900b3280 con 0x7fc3900cce90 2024-03-28T15:01:40.078 INFO:teuthology.orchestra.run.smithi186.stdout:55834574856 2024-03-28T15:01:40.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 -- 172.21.15.186:0/4107138708 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f576c06c600 msgr2=0x7f576c06eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f576c06c600 0x7f576c06eac0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5788154740 tx=0x7f5780009040 comp rx=0 tx=0).stop 2024-03-28T15:01:40.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 -- 172.21.15.186:0/4107138708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57880b39c0 msgr2=0x7f578815dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57880b39c0 0x7f578815dce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5784009890 tx=0x7f57840046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 -- 172.21.15.186:0/4107138708 shutdown_connections 2024-03-28T15:01:40.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f576c06c600 0x7f576c06eac0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5788161da0 0x7f578815ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5788155470 0x7f5788161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 --2- 172.21.15.186:0/4107138708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57880b39c0 0x7f578815dce0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 -- 172.21.15.186:0/4107138708 >> 172.21.15.186:0/4107138708 conn(0x7f578806b640 msgr2=0x7f5788152830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 -- 172.21.15.186:0/4107138708 shutdown_connections 2024-03-28T15:01:40.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.078+0000 7f5790113640 1 -- 172.21.15.186:0/4107138708 wait complete. 2024-03-28T15:01:40.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 -- 172.21.15.186:0/138999098 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc37406c560 msgr2=0x7fc37406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc37406c560 0x7fc37406ea20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc3800017c0 tx=0x7fc380009040 comp rx=0 tx=0).stop 2024-03-28T15:01:40.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 -- 172.21.15.186:0/138999098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3900cce90 msgr2=0x7fc39014b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3900cce90 0x7fc39014b6f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00e9b0 tx=0x7fc38c00ee70 comp rx=0 tx=0).stop 2024-03-28T15:01:40.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 -- 172.21.15.186:0/138999098 shutdown_connections 2024-03-28T15:01:40.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc37406c560 0x7fc37406ea20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc39015b2c0 0x7fc390159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390154a10 0x7fc3900ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 --2- 172.21.15.186:0/138999098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3900cce90 0x7fc39014b6f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 -- 172.21.15.186:0/138999098 >> 172.21.15.186:0/138999098 conn(0x7fc39006b640 msgr2=0x7fc3901515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 -- 172.21.15.186:0/138999098 shutdown_connections 2024-03-28T15:01:40.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.080+0000 7fc397623640 1 -- 172.21.15.186:0/138999098 wait complete. 2024-03-28T15:01:40.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.082+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fc148156720 con 0x7fc1480c1e30 2024-03-28T15:01:40.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.082+0000 7fc12ffff640 1 -- 172.21.15.186:0/175225074 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc148153260 con 0x7fc1480c1e30 2024-03-28T15:01:40.086 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2024-03-28T15:01:40.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc13406c600 msgr2=0x7fc13406eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc13406c600 0x7fc13406eac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc13c007fd0 tx=0x7fc13c007d80 comp rx=0 tx=0).stop 2024-03-28T15:01:40.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1480c1e30 msgr2=0x7fc148072cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1480c1e30 0x7fc148072cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc13800d800 tx=0x7fc13800dcc0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 shutdown_connections 2024-03-28T15:01:40.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc13406c600 0x7fc13406eac0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1480c1e30 0x7fc148072cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc14806f230 0x7fc1480722e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 --2- 172.21.15.186:0/175225074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc14806ee50 0x7fc14814eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 >> 172.21.15.186:0/175225074 conn(0x7fc14806c010 msgr2=0x7fc14814bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 shutdown_connections 2024-03-28T15:01:40.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.084+0000 7fc14e0df640 1 -- 172.21.15.186:0/175225074 wait complete. 2024-03-28T15:01:40.092 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2024-03-28T15:01:40.092 DEBUG:teuthology.parallel:result is None 2024-03-28T15:01:40.094 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-03-28T15:01:40.094 DEBUG:teuthology.parallel:result is None 2024-03-28T15:01:40.097 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2024-03-28T15:01:40.097 DEBUG:teuthology.parallel:result is None 2024-03-28T15:01:40.097 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-28T15:01:40.097 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:01:40.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.238+0000 7fe3c85b5640 1 Processor -- start 2024-03-28T15:01:40.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.238+0000 7fe3c85b5640 1 -- start start 2024-03-28T15:01:40.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.239+0000 7fe3c85b5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe3c00caa70 0x7fe3c00c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.239+0000 7fe3c85b5640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe3c00c6a00 0x7fe3c00cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.239+0000 7fe3c85b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3c00c5380 0x7fe3c00cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.239+0000 7fe3c85b5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fe3c0071a80 con 0x7fe3c00caa70 2024-03-28T15:01:40.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.239+0000 7fe3c85b5640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fe3c00c6030 con 0x7fe3c00c6a00 2024-03-28T15:01:40.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.239+0000 7fe3c85b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c00c6f90 con 0x7fe3c00c5380 2024-03-28T15:01:40.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.239+0000 7fe3c6351640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe3c00caa70 0x7fe3c00c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.239+0000 7fe3c6351640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe3c00caa70 0x7fe3c00c64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48404/0 (socket says 172.21.15.186:48404) 2024-03-28T15:01:40.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.239+0000 7fe3c6351640 1 -- 172.21.15.186:0/464175188 learned_addr learned my addr 172.21.15.186:0/464175188 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:40.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.240+0000 7fe3c6b52640 1 --2- 172.21.15.186:0/464175188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3c00c5380 0x7fe3c00cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.240+0000 7fe3c5b50640 1 --2- 172.21.15.186:0/464175188 >> v2:172.21.15.191:3300/0 conn(0x7fe3c00c6a00 0x7fe3c00cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.240+0000 7fe3c6351640 1 -- 172.21.15.186:0/464175188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3c00c5380 msgr2=0x7fe3c00cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:40.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.240+0000 7fe3c6351640 1 --2- 172.21.15.186:0/464175188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3c00c5380 0x7fe3c00cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.240+0000 7fe3c6351640 1 -- 172.21.15.186:0/464175188 >> v2:172.21.15.191:3300/0 conn(0x7fe3c00c6a00 msgr2=0x7fe3c00cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:40.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.240+0000 7fe3c6351640 1 --2- 172.21.15.186:0/464175188 >> v2:172.21.15.191:3300/0 conn(0x7fe3c00c6a00 0x7fe3c00cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.240+0000 7fe3c6351640 1 -- 172.21.15.186:0/464175188 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c00cc970 con 0x7fe3c00caa70 2024-03-28T15:01:40.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.240+0000 7fe3c6b52640 1 --2- 172.21.15.186:0/464175188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3c00c5380 0x7fe3c00cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:40.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.240+0000 7fe3c5b50640 1 --2- 172.21.15.186:0/464175188 >> v2:172.21.15.191:3300/0 conn(0x7fe3c00c6a00 0x7fe3c00cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:40.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.240+0000 7fe3c6351640 1 --2- 172.21.15.186:0/464175188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c00caa70 0x7fe3c00c64c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe3c015efb0 tx=0x7fe3bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6be52109705e917 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:40.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c534f640 1 -- 172.21.15.186:0/464175188 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc02ca70 con 0x7fe3c00caa70 2024-03-28T15:01:40.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c534f640 1 -- 172.21.15.186:0/464175188 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3bc02cc10 con 0x7fe3c00caa70 2024-03-28T15:01:40.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c534f640 1 -- 172.21.15.186:0/464175188 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc0343f0 con 0x7fe3c00caa70 2024-03-28T15:01:40.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c85b5640 1 -- 172.21.15.186:0/464175188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c00caa70 msgr2=0x7fe3c00c64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c85b5640 1 --2- 172.21.15.186:0/464175188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c00caa70 0x7fe3c00c64c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe3c015efb0 tx=0x7fe3bc02bed0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c85b5640 1 -- 172.21.15.186:0/464175188 shutdown_connections 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c85b5640 1 --2- 172.21.15.186:0/464175188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3c00c5380 0x7fe3c00cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c85b5640 1 --2- 172.21.15.186:0/464175188 >> v2:172.21.15.191:3300/0 conn(0x7fe3c00c6a00 0x7fe3c00cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c85b5640 1 --2- 172.21.15.186:0/464175188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c00caa70 0x7fe3c00c64c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c85b5640 1 -- 172.21.15.186:0/464175188 >> 172.21.15.186:0/464175188 conn(0x7fe3c00d00f0 msgr2=0x7fe3c00d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c85b5640 1 -- 172.21.15.186:0/464175188 shutdown_connections 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.241+0000 7fe3c85b5640 1 -- 172.21.15.186:0/464175188 wait complete. 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.242+0000 7fe3c85b5640 1 Processor -- start 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.242+0000 7fe3c85b5640 1 -- start start 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c85b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3c00c5380 0x7fe3c014e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c85b5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c00c6a00 0x7fe3c00c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c85b5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3c00caa70 0x7fe3c00c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c85b5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c015b530 con 0x7fe3c00c6a00 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c85b5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c00cc690 con 0x7fe3c00caa70 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c85b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c00c5cc0 con 0x7fe3c00c5380 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c5b50640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c00c6a00 0x7fe3c00c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c5b50640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c00c6a00 0x7fe3c00c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48418/0 (socket says 172.21.15.186:48418) 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c5b50640 1 -- 172.21.15.186:0/3711012382 learned_addr learned my addr 172.21.15.186:0/3711012382 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c6b52640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3c00caa70 0x7fe3c00c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c6351640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3c00c5380 0x7fe3c014e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c5b50640 1 -- 172.21.15.186:0/3711012382 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3c00c5380 msgr2=0x7fe3c014e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c5b50640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3c00c5380 0x7fe3c014e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c5b50640 1 -- 172.21.15.186:0/3711012382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3c00caa70 msgr2=0x7fe3c00c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c5b50640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3c00caa70 0x7fe3c00c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.243+0000 7fe3c5b50640 1 -- 172.21.15.186:0/3711012382 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c014e8e0 con 0x7fe3c00c6a00 2024-03-28T15:01:40.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.244+0000 7fe3c5b50640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c00c6a00 0x7fe3c00c8ff0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000c820 tx=0x7fe3b000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:40.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.244+0000 7fe3b6ffd640 1 -- 172.21.15.186:0/3711012382 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0004230 con 0x7fe3c00c6a00 2024-03-28T15:01:40.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.244+0000 7fe3b6ffd640 1 -- 172.21.15.186:0/3711012382 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3b00043d0 con 0x7fe3c00c6a00 2024-03-28T15:01:40.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.244+0000 7fe3c85b5640 1 -- 172.21.15.186:0/3711012382 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c014ebc0 con 0x7fe3c00c6a00 2024-03-28T15:01:40.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.244+0000 7fe3b6ffd640 1 -- 172.21.15.186:0/3711012382 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0007a70 con 0x7fe3c00c6a00 2024-03-28T15:01:40.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.244+0000 7fe3c85b5640 1 -- 172.21.15.186:0/3711012382 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c014f180 con 0x7fe3c00c6a00 2024-03-28T15:01:40.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.245+0000 7fe3c85b5640 1 -- 172.21.15.186:0/3711012382 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c00686d0 con 0x7fe3c00c6a00 2024-03-28T15:01:40.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.245+0000 7fe3b6ffd640 1 -- 172.21.15.186:0/3711012382 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe3b0007c10 con 0x7fe3c00c6a00 2024-03-28T15:01:40.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.245+0000 7fe3b6ffd640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe3a406c560 0x7fe3a406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.246+0000 7fe3b6ffd640 1 -- 172.21.15.186:0/3711012382 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe3b008b7d0 con 0x7fe3c00c6a00 2024-03-28T15:01:40.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.246+0000 7fe3c6351640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe3a406c560 0x7fe3a406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.246+0000 7fe3c6351640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe3a406c560 0x7fe3a406ea20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc035000 tx=0x7fe3bc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:40.250 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.250+0000 7fe3b6ffd640 1 -- 172.21.15.186:0/3711012382 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fe3b0015020 con 0x7fe3c00c6a00 2024-03-28T15:01:40.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.423+0000 7fe3c85b5640 1 -- 172.21.15.186:0/3711012382 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe3c00b2c80 con 0x7fe3a406c560 2024-03-28T15:01:40.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.429+0000 7fe3b6ffd640 1 -- 172.21.15.186:0/3711012382 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251947 (secure 0 0 0) 0x7fe3c00b2c80 con 0x7fe3a406c560 2024-03-28T15:01:40.432 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:40.433 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:01:40.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 -- 172.21.15.186:0/3711012382 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe3a406c560 msgr2=0x7fe3a406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe3a406c560 0x7fe3a406ea20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc035000 tx=0x7fe3bc030000 comp rx=0 tx=0).stop 2024-03-28T15:01:40.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 -- 172.21.15.186:0/3711012382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c00c6a00 msgr2=0x7fe3c00c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c00c6a00 0x7fe3c00c8ff0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000c820 tx=0x7fe3b000cce0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 -- 172.21.15.186:0/3711012382 shutdown_connections 2024-03-28T15:01:40.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe3a406c560 0x7fe3a406ea20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3c00caa70 0x7fe3c00c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c00c6a00 0x7fe3c00c8ff0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 --2- 172.21.15.186:0/3711012382 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3c00c5380 0x7fe3c014e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 -- 172.21.15.186:0/3711012382 >> 172.21.15.186:0/3711012382 conn(0x7fe3c00d00f0 msgr2=0x7fe3c0153d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 -- 172.21.15.186:0/3711012382 shutdown_connections 2024-03-28T15:01:40.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.432+0000 7fe3c85b5640 1 -- 172.21.15.186:0/3711012382 wait complete. 2024-03-28T15:01:40.443 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-28T15:01:38.636292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218948,"kb_used_data":2812,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755772,"statfs":{"total":773094113280,"available":772869910528,"internally_reserved":0,"allocated":2879488,"data_stored":1751416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.560569"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461304+0000","last_change":"2024-03-28T15:01:32.122226+0000","last_active":"2024-03-28T15:01:33.461304+0000","last_peered":"2024-03-28T15:01:33.461304+0000","last_clean":"2024-03-28T15:01:33.461304+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:01:33.461304+0000","last_undegraded":"2024-03-28T15:01:33.461304+0000","last_fullsized":"2024-03-28T15:01:33.461304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104984+0000","last_change":"2024-03-28T15:01:32.115850+0000","last_active":"2024-03-28T15:01:33.104984+0000","last_peered":"2024-03-28T15:01:33.104984+0000","last_clean":"2024-03-28T15:01:33.104984+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:01:33.104984+0000","last_undegraded":"2024-03-28T15:01:33.104984+0000","last_fullsized":"2024-03-28T15:01:33.104984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.119744+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:57.022614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107952+0000","last_change":"2024-03-28T15:01:32.120916+0000","last_active":"2024-03-28T15:01:33.107952+0000","last_peered":"2024-03-28T15:01:33.107952+0000","last_clean":"2024-03-28T15:01:33.107952+0000","last_became_active":"2024-03-28T15:01:32.117449+0000","last_became_peered":"2024-03-28T15:01:32.117449+0000","last_unstale":"2024-03-28T15:01:33.107952+0000","last_undegraded":"2024-03-28T15:01:33.107952+0000","last_fullsized":"2024-03-28T15:01:33.107952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461016+0000","last_change":"2024-03-28T15:01:32.112835+0000","last_active":"2024-03-28T15:01:33.461016+0000","last_peered":"2024-03-28T15:01:33.461016+0000","last_clean":"2024-03-28T15:01:33.461016+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:01:33.461016+0000","last_undegraded":"2024-03-28T15:01:33.461016+0000","last_fullsized":"2024-03-28T15:01:33.461016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107714+0000","last_change":"2024-03-28T15:01:32.116248+0000","last_active":"2024-03-28T15:01:33.107714+0000","last_peered":"2024-03-28T15:01:33.107714+0000","last_clean":"2024-03-28T15:01:33.107714+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:01:33.107714+0000","last_undegraded":"2024-03-28T15:01:33.107714+0000","last_fullsized":"2024-03-28T15:01:33.107714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460486+0000","last_change":"2024-03-28T15:01:32.115221+0000","last_active":"2024-03-28T15:01:33.460486+0000","last_peered":"2024-03-28T15:01:33.460486+0000","last_clean":"2024-03-28T15:01:33.460486+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:01:33.460486+0000","last_undegraded":"2024-03-28T15:01:33.460486+0000","last_fullsized":"2024-03-28T15:01:33.460486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460218+0000","last_change":"2024-03-28T15:01:32.118310+0000","last_active":"2024-03-28T15:01:33.460218+0000","last_peered":"2024-03-28T15:01:33.460218+0000","last_clean":"2024-03-28T15:01:33.460218+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:01:33.460218+0000","last_undegraded":"2024-03-28T15:01:33.460218+0000","last_fullsized":"2024-03-28T15:01:33.460218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461212+0000","last_change":"2024-03-28T15:01:32.114205+0000","last_active":"2024-03-28T15:01:33.461212+0000","last_peered":"2024-03-28T15:01:33.461212+0000","last_clean":"2024-03-28T15:01:33.461212+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:01:33.461212+0000","last_undegraded":"2024-03-28T15:01:33.461212+0000","last_fullsized":"2024-03-28T15:01:33.461212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460905+0000","last_change":"2024-03-28T15:01:32.125178+0000","last_active":"2024-03-28T15:01:33.460905+0000","last_peered":"2024-03-28T15:01:33.460905+0000","last_clean":"2024-03-28T15:01:33.460905+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:01:33.460905+0000","last_undegraded":"2024-03-28T15:01:33.460905+0000","last_fullsized":"2024-03-28T15:01:33.460905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460676+0000","last_change":"2024-03-28T15:01:32.125839+0000","last_active":"2024-03-28T15:01:33.460676+0000","last_peered":"2024-03-28T15:01:33.460676+0000","last_clean":"2024-03-28T15:01:33.460676+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:01:33.460676+0000","last_undegraded":"2024-03-28T15:01:33.460676+0000","last_fullsized":"2024-03-28T15:01:33.460676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107231+0000","last_change":"2024-03-28T15:01:32.121874+0000","last_active":"2024-03-28T15:01:33.107231+0000","last_peered":"2024-03-28T15:01:33.107231+0000","last_clean":"2024-03-28T15:01:33.107231+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:01:33.107231+0000","last_undegraded":"2024-03-28T15:01:33.107231+0000","last_fullsized":"2024-03-28T15:01:33.107231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:20.744143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460735+0000","last_change":"2024-03-28T15:01:32.119348+0000","last_active":"2024-03-28T15:01:33.460735+0000","last_peered":"2024-03-28T15:01:33.460735+0000","last_clean":"2024-03-28T15:01:33.460735+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:01:33.460735+0000","last_undegraded":"2024-03-28T15:01:33.460735+0000","last_fullsized":"2024-03-28T15:01:33.460735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461204+0000","last_change":"2024-03-28T15:01:32.114290+0000","last_active":"2024-03-28T15:01:33.461204+0000","last_peered":"2024-03-28T15:01:33.461204+0000","last_clean":"2024-03-28T15:01:33.461204+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:01:33.461204+0000","last_undegraded":"2024-03-28T15:01:33.461204+0000","last_fullsized":"2024-03-28T15:01:33.461204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105071+0000","last_change":"2024-03-28T15:01:32.113654+0000","last_active":"2024-03-28T15:01:33.105071+0000","last_peered":"2024-03-28T15:01:33.105071+0000","last_clean":"2024-03-28T15:01:33.105071+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:01:33.105071+0000","last_undegraded":"2024-03-28T15:01:33.105071+0000","last_fullsized":"2024-03-28T15:01:33.105071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107620+0000","last_change":"2024-03-28T15:01:32.115901+0000","last_active":"2024-03-28T15:01:33.107620+0000","last_peered":"2024-03-28T15:01:33.107620+0000","last_clean":"2024-03-28T15:01:33.107620+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:01:33.107620+0000","last_undegraded":"2024-03-28T15:01:33.107620+0000","last_fullsized":"2024-03-28T15:01:33.107620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108153+0000","last_change":"2024-03-28T15:01:32.113156+0000","last_active":"2024-03-28T15:01:33.108153+0000","last_peered":"2024-03-28T15:01:33.108153+0000","last_clean":"2024-03-28T15:01:33.108153+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:01:33.108153+0000","last_undegraded":"2024-03-28T15:01:33.108153+0000","last_fullsized":"2024-03-28T15:01:33.108153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460557+0000","last_change":"2024-03-28T15:01:32.119785+0000","last_active":"2024-03-28T15:01:33.460557+0000","last_peered":"2024-03-28T15:01:33.460557+0000","last_clean":"2024-03-28T15:01:33.460557+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:01:33.460557+0000","last_undegraded":"2024-03-28T15:01:33.460557+0000","last_fullsized":"2024-03-28T15:01:33.460557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461426+0000","last_change":"2024-03-28T15:01:32.111912+0000","last_active":"2024-03-28T15:01:33.461426+0000","last_peered":"2024-03-28T15:01:33.461426+0000","last_clean":"2024-03-28T15:01:33.461426+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:01:33.461426+0000","last_undegraded":"2024-03-28T15:01:33.461426+0000","last_fullsized":"2024-03-28T15:01:33.461426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:52.632729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461025+0000","last_change":"2024-03-28T15:01:32.124381+0000","last_active":"2024-03-28T15:01:33.461025+0000","last_peered":"2024-03-28T15:01:33.461025+0000","last_clean":"2024-03-28T15:01:33.461025+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:01:33.461025+0000","last_undegraded":"2024-03-28T15:01:33.461025+0000","last_fullsized":"2024-03-28T15:01:33.461025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.120110+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107929+0000","last_change":"2024-03-28T15:01:32.113699+0000","last_active":"2024-03-28T15:01:33.107929+0000","last_peered":"2024-03-28T15:01:33.107929+0000","last_clean":"2024-03-28T15:01:33.107929+0000","last_became_active":"2024-03-28T15:01:32.113321+0000","last_became_peered":"2024-03-28T15:01:32.113321+0000","last_unstale":"2024-03-28T15:01:33.107929+0000","last_undegraded":"2024-03-28T15:01:33.107929+0000","last_fullsized":"2024-03-28T15:01:33.107929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460506+0000","last_change":"2024-03-28T15:01:32.113345+0000","last_active":"2024-03-28T15:01:33.460506+0000","last_peered":"2024-03-28T15:01:33.460506+0000","last_clean":"2024-03-28T15:01:33.460506+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:01:33.460506+0000","last_undegraded":"2024-03-28T15:01:33.460506+0000","last_fullsized":"2024-03-28T15:01:33.460506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107235+0000","last_change":"2024-03-28T15:01:32.112328+0000","last_active":"2024-03-28T15:01:33.107235+0000","last_peered":"2024-03-28T15:01:33.107235+0000","last_clean":"2024-03-28T15:01:33.107235+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:01:33.107235+0000","last_undegraded":"2024-03-28T15:01:33.107235+0000","last_fullsized":"2024-03-28T15:01:33.107235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461017+0000","last_change":"2024-03-28T15:01:32.122399+0000","last_active":"2024-03-28T15:01:33.461017+0000","last_peered":"2024-03-28T15:01:33.461017+0000","last_clean":"2024-03-28T15:01:33.461017+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:01:33.461017+0000","last_undegraded":"2024-03-28T15:01:33.461017+0000","last_fullsized":"2024-03-28T15:01:33.461017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460971+0000","last_change":"2024-03-28T15:01:32.123879+0000","last_active":"2024-03-28T15:01:33.460971+0000","last_peered":"2024-03-28T15:01:33.460971+0000","last_clean":"2024-03-28T15:01:33.460971+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:01:33.460971+0000","last_undegraded":"2024-03-28T15:01:33.460971+0000","last_fullsized":"2024-03-28T15:01:33.460971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104923+0000","last_change":"2024-03-28T15:01:32.110380+0000","last_active":"2024-03-28T15:01:33.104923+0000","last_peered":"2024-03-28T15:01:33.104923+0000","last_clean":"2024-03-28T15:01:33.104923+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:01:33.104923+0000","last_undegraded":"2024-03-28T15:01:33.104923+0000","last_fullsized":"2024-03-28T15:01:33.104923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460754+0000","last_change":"2024-03-28T15:01:32.113422+0000","last_active":"2024-03-28T15:01:33.460754+0000","last_peered":"2024-03-28T15:01:33.460754+0000","last_clean":"2024-03-28T15:01:33.460754+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:01:33.460754+0000","last_undegraded":"2024-03-28T15:01:33.460754+0000","last_fullsized":"2024-03-28T15:01:33.460754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.117560+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108157+0000","last_change":"2024-03-28T15:01:32.117440+0000","last_active":"2024-03-28T15:01:33.108157+0000","last_peered":"2024-03-28T15:01:33.108157+0000","last_clean":"2024-03-28T15:01:33.108157+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:01:33.108157+0000","last_undegraded":"2024-03-28T15:01:33.108157+0000","last_fullsized":"2024-03-28T15:01:33.108157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460388+0000","last_change":"2024-03-28T15:01:32.111344+0000","last_active":"2024-03-28T15:01:33.460388+0000","last_peered":"2024-03-28T15:01:33.460388+0000","last_clean":"2024-03-28T15:01:33.460388+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:01:33.460388+0000","last_undegraded":"2024-03-28T15:01:33.460388+0000","last_fullsized":"2024-03-28T15:01:33.460388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460295+0000","last_change":"2024-03-28T15:01:32.126811+0000","last_active":"2024-03-28T15:01:33.460295+0000","last_peered":"2024-03-28T15:01:33.460295+0000","last_clean":"2024-03-28T15:01:33.460295+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:01:33.460295+0000","last_undegraded":"2024-03-28T15:01:33.460295+0000","last_fullsized":"2024-03-28T15:01:33.460295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106843+0000","last_change":"2024-03-28T15:01:32.112050+0000","last_active":"2024-03-28T15:01:33.106843+0000","last_peered":"2024-03-28T15:01:33.106843+0000","last_clean":"2024-03-28T15:01:33.106843+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:01:33.106843+0000","last_undegraded":"2024-03-28T15:01:33.106843+0000","last_fullsized":"2024-03-28T15:01:33.106843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:30.376545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.475527+0000","last_change":"2024-03-28T15:01:31.097415+0000","last_active":"2024-03-28T15:01:34.475527+0000","last_peered":"2024-03-28T15:01:34.475527+0000","last_clean":"2024-03-28T15:01:34.475527+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:01:34.475527+0000","last_undegraded":"2024-03-28T15:01:34.475527+0000","last_fullsized":"2024-03-28T15:01:34.475527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461379+0000","last_change":"2024-03-28T15:01:32.123479+0000","last_active":"2024-03-28T15:01:33.461379+0000","last_peered":"2024-03-28T15:01:33.461379+0000","last_clean":"2024-03-28T15:01:33.461379+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:01:33.461379+0000","last_undegraded":"2024-03-28T15:01:33.461379+0000","last_fullsized":"2024-03-28T15:01:33.461379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.476969+0000","last_change":"2024-03-28T15:01:31.088437+0000","last_active":"2024-03-28T15:01:34.476969+0000","last_peered":"2024-03-28T15:01:34.476969+0000","last_clean":"2024-03-28T15:01:34.476969+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:01:34.476969+0000","last_undegraded":"2024-03-28T15:01:34.476969+0000","last_fullsized":"2024-03-28T15:01:34.476969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460403+0000","last_change":"2024-03-28T15:01:31.097499+0000","last_active":"2024-03-28T15:01:33.460403+0000","last_peered":"2024-03-28T15:01:33.460403+0000","last_clean":"2024-03-28T15:01:33.460403+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:01:33.460403+0000","last_undegraded":"2024-03-28T15:01:33.460403+0000","last_fullsized":"2024-03-28T15:01:33.460403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460343+0000","last_change":"2024-03-28T15:01:32.115727+0000","last_active":"2024-03-28T15:01:33.460343+0000","last_peered":"2024-03-28T15:01:33.460343+0000","last_clean":"2024-03-28T15:01:33.460343+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:01:33.460343+0000","last_undegraded":"2024-03-28T15:01:33.460343+0000","last_fullsized":"2024-03-28T15:01:33.460343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107881+0000","last_change":"2024-03-28T15:01:31.086252+0000","last_active":"2024-03-28T15:01:33.107881+0000","last_peered":"2024-03-28T15:01:33.107881+0000","last_clean":"2024-03-28T15:01:33.107881+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:01:33.107881+0000","last_undegraded":"2024-03-28T15:01:33.107881+0000","last_fullsized":"2024-03-28T15:01:33.107881+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104868+0000","last_change":"2024-03-28T15:01:32.110465+0000","last_active":"2024-03-28T15:01:33.104868+0000","last_peered":"2024-03-28T15:01:33.104868+0000","last_clean":"2024-03-28T15:01:33.104868+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:01:33.104868+0000","last_undegraded":"2024-03-28T15:01:33.104868+0000","last_fullsized":"2024-03-28T15:01:33.104868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460446+0000","last_change":"2024-03-28T15:01:31.092338+0000","last_active":"2024-03-28T15:01:33.460446+0000","last_peered":"2024-03-28T15:01:33.460446+0000","last_clean":"2024-03-28T15:01:33.460446+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:01:33.460446+0000","last_undegraded":"2024-03-28T15:01:33.460446+0000","last_fullsized":"2024-03-28T15:01:33.460446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107369+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.107369+0000","last_peered":"2024-03-28T15:01:33.107369+0000","last_clean":"2024-03-28T15:01:33.107369+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:01:33.107369+0000","last_undegraded":"2024-03-28T15:01:33.107369+0000","last_fullsized":"2024-03-28T15:01:33.107369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107482+0000","last_change":"2024-03-28T15:01:31.088909+0000","last_active":"2024-03-28T15:01:33.107482+0000","last_peered":"2024-03-28T15:01:33.107482+0000","last_clean":"2024-03-28T15:01:33.107482+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:01:33.107482+0000","last_undegraded":"2024-03-28T15:01:33.107482+0000","last_fullsized":"2024-03-28T15:01:33.107482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461141+0000","last_change":"2024-03-28T15:01:32.117188+0000","last_active":"2024-03-28T15:01:33.461141+0000","last_peered":"2024-03-28T15:01:33.461141+0000","last_clean":"2024-03-28T15:01:33.461141+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:01:33.461141+0000","last_undegraded":"2024-03-28T15:01:33.461141+0000","last_fullsized":"2024-03-28T15:01:33.461141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.121351+0000","last_change":"2024-03-28T15:01:31.095935+0000","last_active":"2024-03-28T15:01:37.121351+0000","last_peered":"2024-03-28T15:01:37.121351+0000","last_clean":"2024-03-28T15:01:37.121351+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:01:37.121351+0000","last_undegraded":"2024-03-28T15:01:37.121351+0000","last_fullsized":"2024-03-28T15:01:37.121351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107512+0000","last_change":"2024-03-28T15:01:32.120677+0000","last_active":"2024-03-28T15:01:33.107512+0000","last_peered":"2024-03-28T15:01:33.107512+0000","last_clean":"2024-03-28T15:01:33.107512+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.107512+0000","last_undegraded":"2024-03-28T15:01:33.107512+0000","last_fullsized":"2024-03-28T15:01:33.107512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.124097+0000","last_change":"2024-03-28T15:01:31.089380+0000","last_active":"2024-03-28T15:01:37.124097+0000","last_peered":"2024-03-28T15:01:37.124097+0000","last_clean":"2024-03-28T15:01:37.124097+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:01:37.124097+0000","last_undegraded":"2024-03-28T15:01:37.124097+0000","last_fullsized":"2024-03-28T15:01:37.124097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.110749+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.110332+0000","last_became_peered":"2024-03-28T15:01:32.110332+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.122627+0000","last_change":"2024-03-28T15:01:31.098172+0000","last_active":"2024-03-28T15:01:37.122627+0000","last_peered":"2024-03-28T15:01:37.122627+0000","last_clean":"2024-03-28T15:01:37.122627+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:01:37.122627+0000","last_undegraded":"2024-03-28T15:01:37.122627+0000","last_fullsized":"2024-03-28T15:01:37.122627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:16.015528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461045+0000","last_change":"2024-03-28T15:01:32.117690+0000","last_active":"2024-03-28T15:01:33.461045+0000","last_peered":"2024-03-28T15:01:33.461045+0000","last_clean":"2024-03-28T15:01:33.461045+0000","last_became_active":"2024-03-28T15:01:32.117412+0000","last_became_peered":"2024-03-28T15:01:32.117412+0000","last_unstale":"2024-03-28T15:01:33.461045+0000","last_undegraded":"2024-03-28T15:01:33.461045+0000","last_fullsized":"2024-03-28T15:01:33.461045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108124+0000","last_change":"2024-03-28T15:01:31.090519+0000","last_active":"2024-03-28T15:01:33.108124+0000","last_peered":"2024-03-28T15:01:33.108124+0000","last_clean":"2024-03-28T15:01:33.108124+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:01:33.108124+0000","last_undegraded":"2024-03-28T15:01:33.108124+0000","last_fullsized":"2024-03-28T15:01:33.108124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460932+0000","last_change":"2024-03-28T15:01:32.119827+0000","last_active":"2024-03-28T15:01:33.460932+0000","last_peered":"2024-03-28T15:01:33.460932+0000","last_clean":"2024-03-28T15:01:33.460932+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:01:33.460932+0000","last_undegraded":"2024-03-28T15:01:33.460932+0000","last_fullsized":"2024-03-28T15:01:33.460932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461234+0000","last_change":"2024-03-28T15:01:31.088364+0000","last_active":"2024-03-28T15:01:33.461234+0000","last_peered":"2024-03-28T15:01:33.461234+0000","last_clean":"2024-03-28T15:01:33.461234+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:01:33.461234+0000","last_undegraded":"2024-03-28T15:01:33.461234+0000","last_fullsized":"2024-03-28T15:01:33.461234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460422+0000","last_change":"2024-03-28T15:01:32.126177+0000","last_active":"2024-03-28T15:01:33.460422+0000","last_peered":"2024-03-28T15:01:33.460422+0000","last_clean":"2024-03-28T15:01:33.460422+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:01:33.460422+0000","last_undegraded":"2024-03-28T15:01:33.460422+0000","last_fullsized":"2024-03-28T15:01:33.460422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126124+0000","last_change":"2024-03-28T15:01:31.094837+0000","last_active":"2024-03-28T15:01:37.126124+0000","last_peered":"2024-03-28T15:01:37.126124+0000","last_clean":"2024-03-28T15:01:37.126124+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:01:37.126124+0000","last_undegraded":"2024-03-28T15:01:37.126124+0000","last_fullsized":"2024-03-28T15:01:37.126124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460675+0000","last_change":"2024-03-28T15:01:32.113803+0000","last_active":"2024-03-28T15:01:33.460675+0000","last_peered":"2024-03-28T15:01:33.460675+0000","last_clean":"2024-03-28T15:01:33.460675+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:01:33.460675+0000","last_undegraded":"2024-03-28T15:01:33.460675+0000","last_fullsized":"2024-03-28T15:01:33.460675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.477351+0000","last_change":"2024-03-28T15:01:31.092455+0000","last_active":"2024-03-28T15:01:34.477351+0000","last_peered":"2024-03-28T15:01:34.477351+0000","last_clean":"2024-03-28T15:01:34.477351+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:01:34.477351+0000","last_undegraded":"2024-03-28T15:01:34.477351+0000","last_fullsized":"2024-03-28T15:01:34.477351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:04.636216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460748+0000","last_change":"2024-03-28T15:01:32.117065+0000","last_active":"2024-03-28T15:01:33.460748+0000","last_peered":"2024-03-28T15:01:33.460748+0000","last_clean":"2024-03-28T15:01:33.460748+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:01:33.460748+0000","last_undegraded":"2024-03-28T15:01:33.460748+0000","last_fullsized":"2024-03-28T15:01:33.460748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.123413+0000","last_change":"2024-03-28T15:01:31.092175+0000","last_active":"2024-03-28T15:01:37.123413+0000","last_peered":"2024-03-28T15:01:37.123413+0000","last_clean":"2024-03-28T15:01:37.123413+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:01:37.123413+0000","last_undegraded":"2024-03-28T15:01:37.123413+0000","last_fullsized":"2024-03-28T15:01:37.123413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461324+0000","last_change":"2024-03-28T15:01:32.114975+0000","last_active":"2024-03-28T15:01:33.461324+0000","last_peered":"2024-03-28T15:01:33.461324+0000","last_clean":"2024-03-28T15:01:33.461324+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:01:33.461324+0000","last_undegraded":"2024-03-28T15:01:33.461324+0000","last_fullsized":"2024-03-28T15:01:33.461324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.482725+0000","last_change":"2024-03-28T15:01:31.097901+0000","last_active":"2024-03-28T15:01:34.482725+0000","last_peered":"2024-03-28T15:01:34.482725+0000","last_clean":"2024-03-28T15:01:34.482725+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:01:34.482725+0000","last_undegraded":"2024-03-28T15:01:34.482725+0000","last_fullsized":"2024-03-28T15:01:34.482725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108151+0000","last_change":"2024-03-28T15:01:32.120646+0000","last_active":"2024-03-28T15:01:33.108151+0000","last_peered":"2024-03-28T15:01:33.108151+0000","last_clean":"2024-03-28T15:01:33.108151+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:01:33.108151+0000","last_undegraded":"2024-03-28T15:01:33.108151+0000","last_fullsized":"2024-03-28T15:01:33.108151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460197+0000","last_change":"2024-03-28T15:01:31.086634+0000","last_active":"2024-03-28T15:01:33.460197+0000","last_peered":"2024-03-28T15:01:33.460197+0000","last_clean":"2024-03-28T15:01:33.460197+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:01:33.460197+0000","last_undegraded":"2024-03-28T15:01:33.460197+0000","last_fullsized":"2024-03-28T15:01:33.460197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461104+0000","last_change":"2024-03-28T15:01:32.112493+0000","last_active":"2024-03-28T15:01:33.461104+0000","last_peered":"2024-03-28T15:01:33.461104+0000","last_clean":"2024-03-28T15:01:33.461104+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:01:33.461104+0000","last_undegraded":"2024-03-28T15:01:33.461104+0000","last_fullsized":"2024-03-28T15:01:33.461104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461052+0000","last_change":"2024-03-28T15:01:20.724163+0000","last_active":"2024-03-28T15:01:33.461052+0000","last_peered":"2024-03-28T15:01:33.461052+0000","last_clean":"2024-03-28T15:01:33.461052+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:01:33.461052+0000","last_undegraded":"2024-03-28T15:01:33.461052+0000","last_fullsized":"2024-03-28T15:01:33.461052+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_clean_scrub_stamp":"2024-03-28T15:01:19.548090+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:15.791893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107058+0000","last_change":"2024-03-28T15:01:31.085453+0000","last_active":"2024-03-28T15:01:33.107058+0000","last_peered":"2024-03-28T15:01:33.107058+0000","last_clean":"2024-03-28T15:01:33.107058+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:01:33.107058+0000","last_undegraded":"2024-03-28T15:01:33.107058+0000","last_fullsized":"2024-03-28T15:01:33.107058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460326+0000","last_change":"2024-03-28T15:01:32.118678+0000","last_active":"2024-03-28T15:01:33.460326+0000","last_peered":"2024-03-28T15:01:33.460326+0000","last_clean":"2024-03-28T15:01:33.460326+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:01:33.460326+0000","last_undegraded":"2024-03-28T15:01:33.460326+0000","last_fullsized":"2024-03-28T15:01:33.460326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461116+0000","last_change":"2024-03-28T15:01:31.095439+0000","last_active":"2024-03-28T15:01:33.461116+0000","last_peered":"2024-03-28T15:01:33.461116+0000","last_clean":"2024-03-28T15:01:33.461116+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:01:33.461116+0000","last_undegraded":"2024-03-28T15:01:33.461116+0000","last_fullsized":"2024-03-28T15:01:33.461116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460849+0000","last_change":"2024-03-28T15:01:32.122655+0000","last_active":"2024-03-28T15:01:33.460849+0000","last_peered":"2024-03-28T15:01:33.460849+0000","last_clean":"2024-03-28T15:01:33.460849+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:01:33.460849+0000","last_undegraded":"2024-03-28T15:01:33.460849+0000","last_fullsized":"2024-03-28T15:01:33.460849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:05.942532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460385+0000","last_change":"2024-03-28T15:01:31.093265+0000","last_active":"2024-03-28T15:01:33.460385+0000","last_peered":"2024-03-28T15:01:33.460385+0000","last_clean":"2024-03-28T15:01:33.460385+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:01:33.460385+0000","last_undegraded":"2024-03-28T15:01:33.460385+0000","last_fullsized":"2024-03-28T15:01:33.460385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104892+0000","last_change":"2024-03-28T15:01:32.108597+0000","last_active":"2024-03-28T15:01:33.104892+0000","last_peered":"2024-03-28T15:01:33.104892+0000","last_clean":"2024-03-28T15:01:33.104892+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:01:33.104892+0000","last_undegraded":"2024-03-28T15:01:33.104892+0000","last_fullsized":"2024-03-28T15:01:33.104892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108209+0000","last_change":"2024-03-28T15:01:31.094750+0000","last_active":"2024-03-28T15:01:33.108209+0000","last_peered":"2024-03-28T15:01:33.108209+0000","last_clean":"2024-03-28T15:01:33.108209+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:01:33.108209+0000","last_undegraded":"2024-03-28T15:01:33.108209+0000","last_fullsized":"2024-03-28T15:01:33.108209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106970+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.106970+0000","last_peered":"2024-03-28T15:01:33.106970+0000","last_clean":"2024-03-28T15:01:33.106970+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:01:33.106970+0000","last_undegraded":"2024-03-28T15:01:33.106970+0000","last_fullsized":"2024-03-28T15:01:33.106970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:24.747778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104874+0000","last_change":"2024-03-28T15:01:31.093706+0000","last_active":"2024-03-28T15:01:33.104874+0000","last_peered":"2024-03-28T15:01:33.104874+0000","last_clean":"2024-03-28T15:01:33.104874+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:01:33.104874+0000","last_undegraded":"2024-03-28T15:01:33.104874+0000","last_fullsized":"2024-03-28T15:01:33.104874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108112+0000","last_change":"2024-03-28T15:01:32.122840+0000","last_active":"2024-03-28T15:01:33.108112+0000","last_peered":"2024-03-28T15:01:33.108112+0000","last_clean":"2024-03-28T15:01:33.108112+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:01:33.108112+0000","last_undegraded":"2024-03-28T15:01:33.108112+0000","last_fullsized":"2024-03-28T15:01:33.108112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461371+0000","last_change":"2024-03-28T15:01:31.095126+0000","last_active":"2024-03-28T15:01:33.461371+0000","last_peered":"2024-03-28T15:01:33.461371+0000","last_clean":"2024-03-28T15:01:33.461371+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:01:33.461371+0000","last_undegraded":"2024-03-28T15:01:33.461371+0000","last_fullsized":"2024-03-28T15:01:33.461371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:21.669535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104885+0000","last_change":"2024-03-28T15:01:32.111925+0000","last_active":"2024-03-28T15:01:33.104885+0000","last_peered":"2024-03-28T15:01:33.104885+0000","last_clean":"2024-03-28T15:01:33.104885+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:01:33.104885+0000","last_undegraded":"2024-03-28T15:01:33.104885+0000","last_fullsized":"2024-03-28T15:01:33.104885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104964+0000","last_change":"2024-03-28T15:01:31.093197+0000","last_active":"2024-03-28T15:01:33.104964+0000","last_peered":"2024-03-28T15:01:33.104964+0000","last_clean":"2024-03-28T15:01:33.104964+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:01:33.104964+0000","last_undegraded":"2024-03-28T15:01:33.104964+0000","last_fullsized":"2024-03-28T15:01:33.104964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461153+0000","last_change":"2024-03-28T15:01:32.115432+0000","last_active":"2024-03-28T15:01:33.461153+0000","last_peered":"2024-03-28T15:01:33.461153+0000","last_clean":"2024-03-28T15:01:33.461153+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:01:33.461153+0000","last_undegraded":"2024-03-28T15:01:33.461153+0000","last_fullsized":"2024-03-28T15:01:33.461153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.125948+0000","last_change":"2024-03-28T15:01:31.088204+0000","last_active":"2024-03-28T15:01:37.125948+0000","last_peered":"2024-03-28T15:01:37.125948+0000","last_clean":"2024-03-28T15:01:37.125948+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:01:37.125948+0000","last_undegraded":"2024-03-28T15:01:37.125948+0000","last_fullsized":"2024-03-28T15:01:37.125948+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461225+0000","last_change":"2024-03-28T15:01:32.114331+0000","last_active":"2024-03-28T15:01:33.461225+0000","last_peered":"2024-03-28T15:01:33.461225+0000","last_clean":"2024-03-28T15:01:33.461225+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:01:33.461225+0000","last_undegraded":"2024-03-28T15:01:33.461225+0000","last_fullsized":"2024-03-28T15:01:33.461225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460969+0000","last_change":"2024-03-28T15:01:31.095371+0000","last_active":"2024-03-28T15:01:33.460969+0000","last_peered":"2024-03-28T15:01:33.460969+0000","last_clean":"2024-03-28T15:01:33.460969+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:01:33.460969+0000","last_undegraded":"2024-03-28T15:01:33.460969+0000","last_fullsized":"2024-03-28T15:01:33.460969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107409+0000","last_change":"2024-03-28T15:01:32.110950+0000","last_active":"2024-03-28T15:01:33.107409+0000","last_peered":"2024-03-28T15:01:33.107409+0000","last_clean":"2024-03-28T15:01:33.107409+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:01:33.107409+0000","last_undegraded":"2024-03-28T15:01:33.107409+0000","last_fullsized":"2024-03-28T15:01:33.107409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105009+0000","last_change":"2024-03-28T15:01:31.094542+0000","last_active":"2024-03-28T15:01:33.105009+0000","last_peered":"2024-03-28T15:01:33.105009+0000","last_clean":"2024-03-28T15:01:33.105009+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:01:33.105009+0000","last_undegraded":"2024-03-28T15:01:33.105009+0000","last_fullsized":"2024-03-28T15:01:33.105009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460466+0000","last_change":"2024-03-28T15:01:32.112158+0000","last_active":"2024-03-28T15:01:33.460466+0000","last_peered":"2024-03-28T15:01:33.460466+0000","last_clean":"2024-03-28T15:01:33.460466+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:01:33.460466+0000","last_undegraded":"2024-03-28T15:01:33.460466+0000","last_fullsized":"2024-03-28T15:01:33.460466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104884+0000","last_change":"2024-03-28T15:01:31.094440+0000","last_active":"2024-03-28T15:01:33.104884+0000","last_peered":"2024-03-28T15:01:33.104884+0000","last_clean":"2024-03-28T15:01:33.104884+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:01:33.104884+0000","last_undegraded":"2024-03-28T15:01:33.104884+0000","last_fullsized":"2024-03-28T15:01:33.104884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460758+0000","last_change":"2024-03-28T15:01:32.115192+0000","last_active":"2024-03-28T15:01:33.460758+0000","last_peered":"2024-03-28T15:01:33.460758+0000","last_clean":"2024-03-28T15:01:33.460758+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:01:33.460758+0000","last_undegraded":"2024-03-28T15:01:33.460758+0000","last_fullsized":"2024-03-28T15:01:33.460758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.125967+0000","last_change":"2024-03-28T15:01:31.096352+0000","last_active":"2024-03-28T15:01:37.125967+0000","last_peered":"2024-03-28T15:01:37.125967+0000","last_clean":"2024-03-28T15:01:37.125967+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:01:37.125967+0000","last_undegraded":"2024-03-28T15:01:37.125967+0000","last_fullsized":"2024-03-28T15:01:37.125967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460555+0000","last_change":"2024-03-28T15:01:32.124700+0000","last_active":"2024-03-28T15:01:33.460555+0000","last_peered":"2024-03-28T15:01:33.460555+0000","last_clean":"2024-03-28T15:01:33.460555+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:01:33.460555+0000","last_undegraded":"2024-03-28T15:01:33.460555+0000","last_fullsized":"2024-03-28T15:01:33.460555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107999+0000","last_change":"2024-03-28T15:01:32.123638+0000","last_active":"2024-03-28T15:01:33.107999+0000","last_peered":"2024-03-28T15:01:33.107999+0000","last_clean":"2024-03-28T15:01:33.107999+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:01:33.107999+0000","last_undegraded":"2024-03-28T15:01:33.107999+0000","last_fullsized":"2024-03-28T15:01:33.107999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.121539+0000","last_change":"2024-03-28T15:01:31.094281+0000","last_active":"2024-03-28T15:01:37.121539+0000","last_peered":"2024-03-28T15:01:37.121539+0000","last_clean":"2024-03-28T15:01:37.121539+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:01:37.121539+0000","last_undegraded":"2024-03-28T15:01:37.121539+0000","last_fullsized":"2024-03-28T15:01:37.121539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107988+0000","last_change":"2024-03-28T15:01:32.112496+0000","last_active":"2024-03-28T15:01:33.107988+0000","last_peered":"2024-03-28T15:01:33.107988+0000","last_clean":"2024-03-28T15:01:33.107988+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:01:33.107988+0000","last_undegraded":"2024-03-28T15:01:33.107988+0000","last_fullsized":"2024-03-28T15:01:33.107988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126177+0000","last_change":"2024-03-28T15:01:31.093773+0000","last_active":"2024-03-28T15:01:37.126177+0000","last_peered":"2024-03-28T15:01:37.126177+0000","last_clean":"2024-03-28T15:01:37.126177+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:01:37.126177+0000","last_undegraded":"2024-03-28T15:01:37.126177+0000","last_fullsized":"2024-03-28T15:01:37.126177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:54.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460976+0000","last_change":"2024-03-28T15:01:32.117275+0000","last_active":"2024-03-28T15:01:33.460976+0000","last_peered":"2024-03-28T15:01:33.460976+0000","last_clean":"2024-03-28T15:01:33.460976+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:01:33.460976+0000","last_undegraded":"2024-03-28T15:01:33.460976+0000","last_fullsized":"2024-03-28T15:01:33.460976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.475554+0000","last_change":"2024-03-28T15:01:31.096725+0000","last_active":"2024-03-28T15:01:34.475554+0000","last_peered":"2024-03-28T15:01:34.475554+0000","last_clean":"2024-03-28T15:01:34.475554+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:01:34.475554+0000","last_undegraded":"2024-03-28T15:01:34.475554+0000","last_fullsized":"2024-03-28T15:01:34.475554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460734+0000","last_change":"2024-03-28T15:01:32.120071+0000","last_active":"2024-03-28T15:01:33.460734+0000","last_peered":"2024-03-28T15:01:33.460734+0000","last_clean":"2024-03-28T15:01:33.460734+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:01:33.460734+0000","last_undegraded":"2024-03-28T15:01:33.460734+0000","last_fullsized":"2024-03-28T15:01:33.460734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126217+0000","last_change":"2024-03-28T15:01:31.095351+0000","last_active":"2024-03-28T15:01:37.126217+0000","last_peered":"2024-03-28T15:01:37.126217+0000","last_clean":"2024-03-28T15:01:37.126217+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:01:37.126217+0000","last_undegraded":"2024-03-28T15:01:37.126217+0000","last_fullsized":"2024-03-28T15:01:37.126217+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:01:40.449 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:01:40.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.584+0000 7f6d2e7a0640 1 Processor -- start 2024-03-28T15:01:40.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.585+0000 7f6d2e7a0640 1 -- start start 2024-03-28T15:01:40.586 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.585+0000 7f6d2e7a0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6d28072910 0x7f6d280d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.586 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.585+0000 7f6d2e7a0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6d280d0af0 0x7f6d280cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.586 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.585+0000 7f6d2e7a0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d280cad10 con 0x7f6d28154b20 2024-03-28T15:01:40.586 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.585+0000 7f6d2e7a0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6d280cca00 con 0x7f6d280d0af0 2024-03-28T15:01:40.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.585+0000 7f6d2e7a0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f6d2815aa90 con 0x7f6d28072910 2024-03-28T15:01:40.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.586+0000 7f6d2df9f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6d280d0af0 0x7f6d280cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.586+0000 7f6d2df9f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6d280d0af0 0x7f6d280cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48420/0 (socket says 172.21.15.186:48420) 2024-03-28T15:01:40.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.586+0000 7f6d2df9f640 1 -- 172.21.15.186:0/2156780820 learned_addr learned my addr 172.21.15.186:0/2156780820 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:40.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.586+0000 7f6d2cf9d640 1 --2- 172.21.15.186:0/2156780820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6d28072910 0x7f6d280d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.586+0000 7f6d2df9f640 1 -- 172.21.15.186:0/2156780820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6d28072910 msgr2=0x7f6d280d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:40.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.586+0000 7f6d2df9f640 1 --2- 172.21.15.186:0/2156780820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6d28072910 0x7f6d280d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.586+0000 7f6d2df9f640 1 -- 172.21.15.186:0/2156780820 >> v1:172.21.15.191:6789/0 conn(0x7f6d28154b20 legacy=0x7f6d280cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.587 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.586+0000 7f6d2df9f640 1 -- 172.21.15.186:0/2156780820 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2815ad70 con 0x7f6d280d0af0 2024-03-28T15:01:40.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d2df9f640 1 --2- 172.21.15.186:0/2156780820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d280d0af0 0x7f6d280cce90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009e50 tx=0x7f6d2002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf37ffc4ee2a9d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:40.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d1bfff640 1 -- 172.21.15.186:0/2156780820 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2002cb90 con 0x7f6d280d0af0 2024-03-28T15:01:40.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d1bfff640 1 -- 172.21.15.186:0/2156780820 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d2002cd30 con 0x7f6d280d0af0 2024-03-28T15:01:40.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d1bfff640 1 -- 172.21.15.186:0/2156780820 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d20034610 con 0x7f6d280d0af0 2024-03-28T15:01:40.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/2156780820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d280d0af0 msgr2=0x7f6d280cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d2e7a0640 1 --2- 172.21.15.186:0/2156780820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d280d0af0 0x7f6d280cce90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009e50 tx=0x7f6d2002bff0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/2156780820 shutdown_connections 2024-03-28T15:01:40.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d2e7a0640 1 --2- 172.21.15.186:0/2156780820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d280d0af0 0x7f6d280cce90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d2e7a0640 1 --2- 172.21.15.186:0/2156780820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6d28072910 0x7f6d280d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/2156780820 >> 172.21.15.186:0/2156780820 conn(0x7f6d280735c0 msgr2=0x7f6d280739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/2156780820 shutdown_connections 2024-03-28T15:01:40.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.587+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/2156780820 wait complete. 2024-03-28T15:01:40.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.588+0000 7f6d2e7a0640 1 Processor -- start 2024-03-28T15:01:40.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2e7a0640 1 -- start start 2024-03-28T15:01:40.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2e7a0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d28072910 0x7f6d2814f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2e7a0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d280d0af0 0x7f6d28152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2e7a0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6d28154b20 0x7f6d2815b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2e7a0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2815b610 con 0x7f6d28072910 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2e7a0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2815b330 con 0x7f6d280d0af0 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2e7a0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f6d2815a760 con 0x7f6d28154b20 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2d79e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d28072910 0x7f6d2814f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2d79e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d28072910 0x7f6d2814f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48434/0 (socket says 172.21.15.186:48434) 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2d79e640 1 -- 172.21.15.186:0/1909021321 learned_addr learned my addr 172.21.15.186:0/1909021321 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2cf9d640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d280d0af0 0x7f6d28152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.589+0000 7f6d2df9f640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6d28154b20 0x7f6d2815b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.590+0000 7f6d2d79e640 1 -- 172.21.15.186:0/1909021321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6d28154b20 msgr2=0x7f6d2815b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.590+0000 7f6d2d79e640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6d28154b20 0x7f6d2815b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.590+0000 7f6d2d79e640 1 -- 172.21.15.186:0/1909021321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d280d0af0 msgr2=0x7f6d28152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.590+0000 7f6d2d79e640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d280d0af0 0x7f6d28152ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.590+0000 7f6d2d79e640 1 -- 172.21.15.186:0/1909021321 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2814fdc0 con 0x7f6d28072910 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.590+0000 7f6d2d79e640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d28072910 0x7f6d2814f880 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00c3c0 tx=0x7f6d1c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.590+0000 7f6d19ffb640 1 -- 172.21.15.186:0/1909021321 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c0052d0 con 0x7f6d28072910 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.590+0000 7f6d19ffb640 1 -- 172.21.15.186:0/1909021321 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d1c005490 con 0x7f6d28072910 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.590+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/1909021321 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2815e8d0 con 0x7f6d28072910 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.590+0000 7f6d19ffb640 1 -- 172.21.15.186:0/1909021321 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c007260 con 0x7f6d28072910 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.590+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/1909021321 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d280b52e0 con 0x7f6d28072910 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.591+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/1909021321 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d28069500 con 0x7f6d28072910 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.591+0000 7f6d19ffb640 1 -- 172.21.15.186:0/1909021321 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6d1c007400 con 0x7f6d28072910 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.592+0000 7f6d19ffb640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6d0c06c530 0x7f6d0c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.592+0000 7f6d19ffb640 1 -- 172.21.15.186:0/1909021321 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6d1c08aad0 con 0x7f6d28072910 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.592+0000 7f6d2cf9d640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6d0c06c530 0x7f6d0c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.592+0000 7f6d2cf9d640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6d0c06c530 0x7f6d0c06e9f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6d28154440 tx=0x7f6d24009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:40.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.596+0000 7f6d19ffb640 1 -- 172.21.15.186:0/1909021321 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6d1c058aa0 con 0x7f6d28072910 2024-03-28T15:01:40.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.770+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/1909021321 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6d2815eb60 con 0x7f6d0c06c530 2024-03-28T15:01:40.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.776+0000 7f6d19ffb640 1 -- 172.21.15.186:0/1909021321 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251835 (secure 0 0 0) 0x7f6d2815eb60 con 0x7f6d0c06c530 2024-03-28T15:01:40.779 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:40.780 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:01:40.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/1909021321 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6d0c06c530 msgr2=0x7f6d0c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6d0c06c530 0x7f6d0c06e9f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6d28154440 tx=0x7f6d24009190 comp rx=0 tx=0).stop 2024-03-28T15:01:40.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/1909021321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d28072910 msgr2=0x7f6d2814f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d28072910 0x7f6d2814f880 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00c3c0 tx=0x7f6d1c00c880 comp rx=0 tx=0).stop 2024-03-28T15:01:40.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/1909021321 shutdown_connections 2024-03-28T15:01:40.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6d0c06c530 0x7f6d0c06e9f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6d28154b20 0x7f6d2815b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d280d0af0 0x7f6d28152ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 --2- 172.21.15.186:0/1909021321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d28072910 0x7f6d2814f880 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/1909021321 >> 172.21.15.186:0/1909021321 conn(0x7f6d280735c0 msgr2=0x7f6d280c5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/1909021321 shutdown_connections 2024-03-28T15:01:40.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.779+0000 7f6d2e7a0640 1 -- 172.21.15.186:0/1909021321 wait complete. 2024-03-28T15:01:40.790 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-28T15:01:40.636757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218952,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755768,"statfs":{"total":773094113280,"available":772869906432,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.537407"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461304+0000","last_change":"2024-03-28T15:01:32.122226+0000","last_active":"2024-03-28T15:01:33.461304+0000","last_peered":"2024-03-28T15:01:33.461304+0000","last_clean":"2024-03-28T15:01:33.461304+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:01:33.461304+0000","last_undegraded":"2024-03-28T15:01:33.461304+0000","last_fullsized":"2024-03-28T15:01:33.461304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104984+0000","last_change":"2024-03-28T15:01:32.115850+0000","last_active":"2024-03-28T15:01:33.104984+0000","last_peered":"2024-03-28T15:01:33.104984+0000","last_clean":"2024-03-28T15:01:33.104984+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:01:33.104984+0000","last_undegraded":"2024-03-28T15:01:33.104984+0000","last_fullsized":"2024-03-28T15:01:33.104984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.119744+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:57.022614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107952+0000","last_change":"2024-03-28T15:01:32.120916+0000","last_active":"2024-03-28T15:01:33.107952+0000","last_peered":"2024-03-28T15:01:33.107952+0000","last_clean":"2024-03-28T15:01:33.107952+0000","last_became_active":"2024-03-28T15:01:32.117449+0000","last_became_peered":"2024-03-28T15:01:32.117449+0000","last_unstale":"2024-03-28T15:01:33.107952+0000","last_undegraded":"2024-03-28T15:01:33.107952+0000","last_fullsized":"2024-03-28T15:01:33.107952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461016+0000","last_change":"2024-03-28T15:01:32.112835+0000","last_active":"2024-03-28T15:01:33.461016+0000","last_peered":"2024-03-28T15:01:33.461016+0000","last_clean":"2024-03-28T15:01:33.461016+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:01:33.461016+0000","last_undegraded":"2024-03-28T15:01:33.461016+0000","last_fullsized":"2024-03-28T15:01:33.461016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107714+0000","last_change":"2024-03-28T15:01:32.116248+0000","last_active":"2024-03-28T15:01:33.107714+0000","last_peered":"2024-03-28T15:01:33.107714+0000","last_clean":"2024-03-28T15:01:33.107714+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:01:33.107714+0000","last_undegraded":"2024-03-28T15:01:33.107714+0000","last_fullsized":"2024-03-28T15:01:33.107714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460486+0000","last_change":"2024-03-28T15:01:32.115221+0000","last_active":"2024-03-28T15:01:33.460486+0000","last_peered":"2024-03-28T15:01:33.460486+0000","last_clean":"2024-03-28T15:01:33.460486+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:01:33.460486+0000","last_undegraded":"2024-03-28T15:01:33.460486+0000","last_fullsized":"2024-03-28T15:01:33.460486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460218+0000","last_change":"2024-03-28T15:01:32.118310+0000","last_active":"2024-03-28T15:01:33.460218+0000","last_peered":"2024-03-28T15:01:33.460218+0000","last_clean":"2024-03-28T15:01:33.460218+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:01:33.460218+0000","last_undegraded":"2024-03-28T15:01:33.460218+0000","last_fullsized":"2024-03-28T15:01:33.460218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461212+0000","last_change":"2024-03-28T15:01:32.114205+0000","last_active":"2024-03-28T15:01:33.461212+0000","last_peered":"2024-03-28T15:01:33.461212+0000","last_clean":"2024-03-28T15:01:33.461212+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:01:33.461212+0000","last_undegraded":"2024-03-28T15:01:33.461212+0000","last_fullsized":"2024-03-28T15:01:33.461212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460905+0000","last_change":"2024-03-28T15:01:32.125178+0000","last_active":"2024-03-28T15:01:33.460905+0000","last_peered":"2024-03-28T15:01:33.460905+0000","last_clean":"2024-03-28T15:01:33.460905+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:01:33.460905+0000","last_undegraded":"2024-03-28T15:01:33.460905+0000","last_fullsized":"2024-03-28T15:01:33.460905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460676+0000","last_change":"2024-03-28T15:01:32.125839+0000","last_active":"2024-03-28T15:01:33.460676+0000","last_peered":"2024-03-28T15:01:33.460676+0000","last_clean":"2024-03-28T15:01:33.460676+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:01:33.460676+0000","last_undegraded":"2024-03-28T15:01:33.460676+0000","last_fullsized":"2024-03-28T15:01:33.460676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107231+0000","last_change":"2024-03-28T15:01:32.121874+0000","last_active":"2024-03-28T15:01:33.107231+0000","last_peered":"2024-03-28T15:01:33.107231+0000","last_clean":"2024-03-28T15:01:33.107231+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:01:33.107231+0000","last_undegraded":"2024-03-28T15:01:33.107231+0000","last_fullsized":"2024-03-28T15:01:33.107231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:20.744143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460735+0000","last_change":"2024-03-28T15:01:32.119348+0000","last_active":"2024-03-28T15:01:33.460735+0000","last_peered":"2024-03-28T15:01:33.460735+0000","last_clean":"2024-03-28T15:01:33.460735+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:01:33.460735+0000","last_undegraded":"2024-03-28T15:01:33.460735+0000","last_fullsized":"2024-03-28T15:01:33.460735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461204+0000","last_change":"2024-03-28T15:01:32.114290+0000","last_active":"2024-03-28T15:01:33.461204+0000","last_peered":"2024-03-28T15:01:33.461204+0000","last_clean":"2024-03-28T15:01:33.461204+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:01:33.461204+0000","last_undegraded":"2024-03-28T15:01:33.461204+0000","last_fullsized":"2024-03-28T15:01:33.461204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105071+0000","last_change":"2024-03-28T15:01:32.113654+0000","last_active":"2024-03-28T15:01:33.105071+0000","last_peered":"2024-03-28T15:01:33.105071+0000","last_clean":"2024-03-28T15:01:33.105071+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:01:33.105071+0000","last_undegraded":"2024-03-28T15:01:33.105071+0000","last_fullsized":"2024-03-28T15:01:33.105071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107620+0000","last_change":"2024-03-28T15:01:32.115901+0000","last_active":"2024-03-28T15:01:33.107620+0000","last_peered":"2024-03-28T15:01:33.107620+0000","last_clean":"2024-03-28T15:01:33.107620+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:01:33.107620+0000","last_undegraded":"2024-03-28T15:01:33.107620+0000","last_fullsized":"2024-03-28T15:01:33.107620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108153+0000","last_change":"2024-03-28T15:01:32.113156+0000","last_active":"2024-03-28T15:01:33.108153+0000","last_peered":"2024-03-28T15:01:33.108153+0000","last_clean":"2024-03-28T15:01:33.108153+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:01:33.108153+0000","last_undegraded":"2024-03-28T15:01:33.108153+0000","last_fullsized":"2024-03-28T15:01:33.108153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460557+0000","last_change":"2024-03-28T15:01:32.119785+0000","last_active":"2024-03-28T15:01:33.460557+0000","last_peered":"2024-03-28T15:01:33.460557+0000","last_clean":"2024-03-28T15:01:33.460557+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:01:33.460557+0000","last_undegraded":"2024-03-28T15:01:33.460557+0000","last_fullsized":"2024-03-28T15:01:33.460557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461426+0000","last_change":"2024-03-28T15:01:32.111912+0000","last_active":"2024-03-28T15:01:33.461426+0000","last_peered":"2024-03-28T15:01:33.461426+0000","last_clean":"2024-03-28T15:01:33.461426+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:01:33.461426+0000","last_undegraded":"2024-03-28T15:01:33.461426+0000","last_fullsized":"2024-03-28T15:01:33.461426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:52.632729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461025+0000","last_change":"2024-03-28T15:01:32.124381+0000","last_active":"2024-03-28T15:01:33.461025+0000","last_peered":"2024-03-28T15:01:33.461025+0000","last_clean":"2024-03-28T15:01:33.461025+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:01:33.461025+0000","last_undegraded":"2024-03-28T15:01:33.461025+0000","last_fullsized":"2024-03-28T15:01:33.461025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.120110+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107929+0000","last_change":"2024-03-28T15:01:32.113699+0000","last_active":"2024-03-28T15:01:33.107929+0000","last_peered":"2024-03-28T15:01:33.107929+0000","last_clean":"2024-03-28T15:01:33.107929+0000","last_became_active":"2024-03-28T15:01:32.113321+0000","last_became_peered":"2024-03-28T15:01:32.113321+0000","last_unstale":"2024-03-28T15:01:33.107929+0000","last_undegraded":"2024-03-28T15:01:33.107929+0000","last_fullsized":"2024-03-28T15:01:33.107929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460506+0000","last_change":"2024-03-28T15:01:32.113345+0000","last_active":"2024-03-28T15:01:33.460506+0000","last_peered":"2024-03-28T15:01:33.460506+0000","last_clean":"2024-03-28T15:01:33.460506+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:01:33.460506+0000","last_undegraded":"2024-03-28T15:01:33.460506+0000","last_fullsized":"2024-03-28T15:01:33.460506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107235+0000","last_change":"2024-03-28T15:01:32.112328+0000","last_active":"2024-03-28T15:01:33.107235+0000","last_peered":"2024-03-28T15:01:33.107235+0000","last_clean":"2024-03-28T15:01:33.107235+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:01:33.107235+0000","last_undegraded":"2024-03-28T15:01:33.107235+0000","last_fullsized":"2024-03-28T15:01:33.107235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461017+0000","last_change":"2024-03-28T15:01:32.122399+0000","last_active":"2024-03-28T15:01:33.461017+0000","last_peered":"2024-03-28T15:01:33.461017+0000","last_clean":"2024-03-28T15:01:33.461017+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:01:33.461017+0000","last_undegraded":"2024-03-28T15:01:33.461017+0000","last_fullsized":"2024-03-28T15:01:33.461017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460971+0000","last_change":"2024-03-28T15:01:32.123879+0000","last_active":"2024-03-28T15:01:33.460971+0000","last_peered":"2024-03-28T15:01:33.460971+0000","last_clean":"2024-03-28T15:01:33.460971+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:01:33.460971+0000","last_undegraded":"2024-03-28T15:01:33.460971+0000","last_fullsized":"2024-03-28T15:01:33.460971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104923+0000","last_change":"2024-03-28T15:01:32.110380+0000","last_active":"2024-03-28T15:01:33.104923+0000","last_peered":"2024-03-28T15:01:33.104923+0000","last_clean":"2024-03-28T15:01:33.104923+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:01:33.104923+0000","last_undegraded":"2024-03-28T15:01:33.104923+0000","last_fullsized":"2024-03-28T15:01:33.104923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460754+0000","last_change":"2024-03-28T15:01:32.113422+0000","last_active":"2024-03-28T15:01:33.460754+0000","last_peered":"2024-03-28T15:01:33.460754+0000","last_clean":"2024-03-28T15:01:33.460754+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:01:33.460754+0000","last_undegraded":"2024-03-28T15:01:33.460754+0000","last_fullsized":"2024-03-28T15:01:33.460754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.117560+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108157+0000","last_change":"2024-03-28T15:01:32.117440+0000","last_active":"2024-03-28T15:01:33.108157+0000","last_peered":"2024-03-28T15:01:33.108157+0000","last_clean":"2024-03-28T15:01:33.108157+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:01:33.108157+0000","last_undegraded":"2024-03-28T15:01:33.108157+0000","last_fullsized":"2024-03-28T15:01:33.108157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460388+0000","last_change":"2024-03-28T15:01:32.111344+0000","last_active":"2024-03-28T15:01:33.460388+0000","last_peered":"2024-03-28T15:01:33.460388+0000","last_clean":"2024-03-28T15:01:33.460388+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:01:33.460388+0000","last_undegraded":"2024-03-28T15:01:33.460388+0000","last_fullsized":"2024-03-28T15:01:33.460388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460295+0000","last_change":"2024-03-28T15:01:32.126811+0000","last_active":"2024-03-28T15:01:33.460295+0000","last_peered":"2024-03-28T15:01:33.460295+0000","last_clean":"2024-03-28T15:01:33.460295+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:01:33.460295+0000","last_undegraded":"2024-03-28T15:01:33.460295+0000","last_fullsized":"2024-03-28T15:01:33.460295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106843+0000","last_change":"2024-03-28T15:01:32.112050+0000","last_active":"2024-03-28T15:01:33.106843+0000","last_peered":"2024-03-28T15:01:33.106843+0000","last_clean":"2024-03-28T15:01:33.106843+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:01:33.106843+0000","last_undegraded":"2024-03-28T15:01:33.106843+0000","last_fullsized":"2024-03-28T15:01:33.106843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:30.376545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.475527+0000","last_change":"2024-03-28T15:01:31.097415+0000","last_active":"2024-03-28T15:01:34.475527+0000","last_peered":"2024-03-28T15:01:34.475527+0000","last_clean":"2024-03-28T15:01:34.475527+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:01:34.475527+0000","last_undegraded":"2024-03-28T15:01:34.475527+0000","last_fullsized":"2024-03-28T15:01:34.475527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461379+0000","last_change":"2024-03-28T15:01:32.123479+0000","last_active":"2024-03-28T15:01:33.461379+0000","last_peered":"2024-03-28T15:01:33.461379+0000","last_clean":"2024-03-28T15:01:33.461379+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:01:33.461379+0000","last_undegraded":"2024-03-28T15:01:33.461379+0000","last_fullsized":"2024-03-28T15:01:33.461379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.476969+0000","last_change":"2024-03-28T15:01:31.088437+0000","last_active":"2024-03-28T15:01:34.476969+0000","last_peered":"2024-03-28T15:01:34.476969+0000","last_clean":"2024-03-28T15:01:34.476969+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:01:34.476969+0000","last_undegraded":"2024-03-28T15:01:34.476969+0000","last_fullsized":"2024-03-28T15:01:34.476969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460403+0000","last_change":"2024-03-28T15:01:31.097499+0000","last_active":"2024-03-28T15:01:33.460403+0000","last_peered":"2024-03-28T15:01:33.460403+0000","last_clean":"2024-03-28T15:01:33.460403+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:01:33.460403+0000","last_undegraded":"2024-03-28T15:01:33.460403+0000","last_fullsized":"2024-03-28T15:01:33.460403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460343+0000","last_change":"2024-03-28T15:01:32.115727+0000","last_active":"2024-03-28T15:01:33.460343+0000","last_peered":"2024-03-28T15:01:33.460343+0000","last_clean":"2024-03-28T15:01:33.460343+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:01:33.460343+0000","last_undegraded":"2024-03-28T15:01:33.460343+0000","last_fullsized":"2024-03-28T15:01:33.460343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107881+0000","last_change":"2024-03-28T15:01:31.086252+0000","last_active":"2024-03-28T15:01:33.107881+0000","last_peered":"2024-03-28T15:01:33.107881+0000","last_clean":"2024-03-28T15:01:33.107881+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:01:33.107881+0000","last_undegraded":"2024-03-28T15:01:33.107881+0000","last_fullsized":"2024-03-28T15:01:33.107881+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104868+0000","last_change":"2024-03-28T15:01:32.110465+0000","last_active":"2024-03-28T15:01:33.104868+0000","last_peered":"2024-03-28T15:01:33.104868+0000","last_clean":"2024-03-28T15:01:33.104868+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:01:33.104868+0000","last_undegraded":"2024-03-28T15:01:33.104868+0000","last_fullsized":"2024-03-28T15:01:33.104868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460446+0000","last_change":"2024-03-28T15:01:31.092338+0000","last_active":"2024-03-28T15:01:33.460446+0000","last_peered":"2024-03-28T15:01:33.460446+0000","last_clean":"2024-03-28T15:01:33.460446+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:01:33.460446+0000","last_undegraded":"2024-03-28T15:01:33.460446+0000","last_fullsized":"2024-03-28T15:01:33.460446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107369+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.107369+0000","last_peered":"2024-03-28T15:01:33.107369+0000","last_clean":"2024-03-28T15:01:33.107369+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:01:33.107369+0000","last_undegraded":"2024-03-28T15:01:33.107369+0000","last_fullsized":"2024-03-28T15:01:33.107369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107482+0000","last_change":"2024-03-28T15:01:31.088909+0000","last_active":"2024-03-28T15:01:33.107482+0000","last_peered":"2024-03-28T15:01:33.107482+0000","last_clean":"2024-03-28T15:01:33.107482+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:01:33.107482+0000","last_undegraded":"2024-03-28T15:01:33.107482+0000","last_fullsized":"2024-03-28T15:01:33.107482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461141+0000","last_change":"2024-03-28T15:01:32.117188+0000","last_active":"2024-03-28T15:01:33.461141+0000","last_peered":"2024-03-28T15:01:33.461141+0000","last_clean":"2024-03-28T15:01:33.461141+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:01:33.461141+0000","last_undegraded":"2024-03-28T15:01:33.461141+0000","last_fullsized":"2024-03-28T15:01:33.461141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.121351+0000","last_change":"2024-03-28T15:01:31.095935+0000","last_active":"2024-03-28T15:01:37.121351+0000","last_peered":"2024-03-28T15:01:37.121351+0000","last_clean":"2024-03-28T15:01:37.121351+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:01:37.121351+0000","last_undegraded":"2024-03-28T15:01:37.121351+0000","last_fullsized":"2024-03-28T15:01:37.121351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107512+0000","last_change":"2024-03-28T15:01:32.120677+0000","last_active":"2024-03-28T15:01:33.107512+0000","last_peered":"2024-03-28T15:01:33.107512+0000","last_clean":"2024-03-28T15:01:33.107512+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.107512+0000","last_undegraded":"2024-03-28T15:01:33.107512+0000","last_fullsized":"2024-03-28T15:01:33.107512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.124097+0000","last_change":"2024-03-28T15:01:31.089380+0000","last_active":"2024-03-28T15:01:37.124097+0000","last_peered":"2024-03-28T15:01:37.124097+0000","last_clean":"2024-03-28T15:01:37.124097+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:01:37.124097+0000","last_undegraded":"2024-03-28T15:01:37.124097+0000","last_fullsized":"2024-03-28T15:01:37.124097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.110749+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.110332+0000","last_became_peered":"2024-03-28T15:01:32.110332+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.122627+0000","last_change":"2024-03-28T15:01:31.098172+0000","last_active":"2024-03-28T15:01:37.122627+0000","last_peered":"2024-03-28T15:01:37.122627+0000","last_clean":"2024-03-28T15:01:37.122627+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:01:37.122627+0000","last_undegraded":"2024-03-28T15:01:37.122627+0000","last_fullsized":"2024-03-28T15:01:37.122627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:16.015528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461045+0000","last_change":"2024-03-28T15:01:32.117690+0000","last_active":"2024-03-28T15:01:33.461045+0000","last_peered":"2024-03-28T15:01:33.461045+0000","last_clean":"2024-03-28T15:01:33.461045+0000","last_became_active":"2024-03-28T15:01:32.117412+0000","last_became_peered":"2024-03-28T15:01:32.117412+0000","last_unstale":"2024-03-28T15:01:33.461045+0000","last_undegraded":"2024-03-28T15:01:33.461045+0000","last_fullsized":"2024-03-28T15:01:33.461045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108124+0000","last_change":"2024-03-28T15:01:31.090519+0000","last_active":"2024-03-28T15:01:33.108124+0000","last_peered":"2024-03-28T15:01:33.108124+0000","last_clean":"2024-03-28T15:01:33.108124+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:01:33.108124+0000","last_undegraded":"2024-03-28T15:01:33.108124+0000","last_fullsized":"2024-03-28T15:01:33.108124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460932+0000","last_change":"2024-03-28T15:01:32.119827+0000","last_active":"2024-03-28T15:01:33.460932+0000","last_peered":"2024-03-28T15:01:33.460932+0000","last_clean":"2024-03-28T15:01:33.460932+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:01:33.460932+0000","last_undegraded":"2024-03-28T15:01:33.460932+0000","last_fullsized":"2024-03-28T15:01:33.460932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461234+0000","last_change":"2024-03-28T15:01:31.088364+0000","last_active":"2024-03-28T15:01:33.461234+0000","last_peered":"2024-03-28T15:01:33.461234+0000","last_clean":"2024-03-28T15:01:33.461234+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:01:33.461234+0000","last_undegraded":"2024-03-28T15:01:33.461234+0000","last_fullsized":"2024-03-28T15:01:33.461234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460422+0000","last_change":"2024-03-28T15:01:32.126177+0000","last_active":"2024-03-28T15:01:33.460422+0000","last_peered":"2024-03-28T15:01:33.460422+0000","last_clean":"2024-03-28T15:01:33.460422+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:01:33.460422+0000","last_undegraded":"2024-03-28T15:01:33.460422+0000","last_fullsized":"2024-03-28T15:01:33.460422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126124+0000","last_change":"2024-03-28T15:01:31.094837+0000","last_active":"2024-03-28T15:01:37.126124+0000","last_peered":"2024-03-28T15:01:37.126124+0000","last_clean":"2024-03-28T15:01:37.126124+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:01:37.126124+0000","last_undegraded":"2024-03-28T15:01:37.126124+0000","last_fullsized":"2024-03-28T15:01:37.126124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460675+0000","last_change":"2024-03-28T15:01:32.113803+0000","last_active":"2024-03-28T15:01:33.460675+0000","last_peered":"2024-03-28T15:01:33.460675+0000","last_clean":"2024-03-28T15:01:33.460675+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:01:33.460675+0000","last_undegraded":"2024-03-28T15:01:33.460675+0000","last_fullsized":"2024-03-28T15:01:33.460675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.477351+0000","last_change":"2024-03-28T15:01:31.092455+0000","last_active":"2024-03-28T15:01:34.477351+0000","last_peered":"2024-03-28T15:01:34.477351+0000","last_clean":"2024-03-28T15:01:34.477351+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:01:34.477351+0000","last_undegraded":"2024-03-28T15:01:34.477351+0000","last_fullsized":"2024-03-28T15:01:34.477351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:04.636216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460748+0000","last_change":"2024-03-28T15:01:32.117065+0000","last_active":"2024-03-28T15:01:33.460748+0000","last_peered":"2024-03-28T15:01:33.460748+0000","last_clean":"2024-03-28T15:01:33.460748+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:01:33.460748+0000","last_undegraded":"2024-03-28T15:01:33.460748+0000","last_fullsized":"2024-03-28T15:01:33.460748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.123413+0000","last_change":"2024-03-28T15:01:31.092175+0000","last_active":"2024-03-28T15:01:37.123413+0000","last_peered":"2024-03-28T15:01:37.123413+0000","last_clean":"2024-03-28T15:01:37.123413+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:01:37.123413+0000","last_undegraded":"2024-03-28T15:01:37.123413+0000","last_fullsized":"2024-03-28T15:01:37.123413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461324+0000","last_change":"2024-03-28T15:01:32.114975+0000","last_active":"2024-03-28T15:01:33.461324+0000","last_peered":"2024-03-28T15:01:33.461324+0000","last_clean":"2024-03-28T15:01:33.461324+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:01:33.461324+0000","last_undegraded":"2024-03-28T15:01:33.461324+0000","last_fullsized":"2024-03-28T15:01:33.461324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.482725+0000","last_change":"2024-03-28T15:01:31.097901+0000","last_active":"2024-03-28T15:01:34.482725+0000","last_peered":"2024-03-28T15:01:34.482725+0000","last_clean":"2024-03-28T15:01:34.482725+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:01:34.482725+0000","last_undegraded":"2024-03-28T15:01:34.482725+0000","last_fullsized":"2024-03-28T15:01:34.482725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108151+0000","last_change":"2024-03-28T15:01:32.120646+0000","last_active":"2024-03-28T15:01:33.108151+0000","last_peered":"2024-03-28T15:01:33.108151+0000","last_clean":"2024-03-28T15:01:33.108151+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:01:33.108151+0000","last_undegraded":"2024-03-28T15:01:33.108151+0000","last_fullsized":"2024-03-28T15:01:33.108151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460197+0000","last_change":"2024-03-28T15:01:31.086634+0000","last_active":"2024-03-28T15:01:33.460197+0000","last_peered":"2024-03-28T15:01:33.460197+0000","last_clean":"2024-03-28T15:01:33.460197+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:01:33.460197+0000","last_undegraded":"2024-03-28T15:01:33.460197+0000","last_fullsized":"2024-03-28T15:01:33.460197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461104+0000","last_change":"2024-03-28T15:01:32.112493+0000","last_active":"2024-03-28T15:01:33.461104+0000","last_peered":"2024-03-28T15:01:33.461104+0000","last_clean":"2024-03-28T15:01:33.461104+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:01:33.461104+0000","last_undegraded":"2024-03-28T15:01:33.461104+0000","last_fullsized":"2024-03-28T15:01:33.461104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461052+0000","last_change":"2024-03-28T15:01:20.724163+0000","last_active":"2024-03-28T15:01:33.461052+0000","last_peered":"2024-03-28T15:01:33.461052+0000","last_clean":"2024-03-28T15:01:33.461052+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:01:33.461052+0000","last_undegraded":"2024-03-28T15:01:33.461052+0000","last_fullsized":"2024-03-28T15:01:33.461052+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_clean_scrub_stamp":"2024-03-28T15:01:19.548090+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:15.791893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107058+0000","last_change":"2024-03-28T15:01:31.085453+0000","last_active":"2024-03-28T15:01:33.107058+0000","last_peered":"2024-03-28T15:01:33.107058+0000","last_clean":"2024-03-28T15:01:33.107058+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:01:33.107058+0000","last_undegraded":"2024-03-28T15:01:33.107058+0000","last_fullsized":"2024-03-28T15:01:33.107058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460326+0000","last_change":"2024-03-28T15:01:32.118678+0000","last_active":"2024-03-28T15:01:33.460326+0000","last_peered":"2024-03-28T15:01:33.460326+0000","last_clean":"2024-03-28T15:01:33.460326+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:01:33.460326+0000","last_undegraded":"2024-03-28T15:01:33.460326+0000","last_fullsized":"2024-03-28T15:01:33.460326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461116+0000","last_change":"2024-03-28T15:01:31.095439+0000","last_active":"2024-03-28T15:01:33.461116+0000","last_peered":"2024-03-28T15:01:33.461116+0000","last_clean":"2024-03-28T15:01:33.461116+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:01:33.461116+0000","last_undegraded":"2024-03-28T15:01:33.461116+0000","last_fullsized":"2024-03-28T15:01:33.461116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460849+0000","last_change":"2024-03-28T15:01:32.122655+0000","last_active":"2024-03-28T15:01:33.460849+0000","last_peered":"2024-03-28T15:01:33.460849+0000","last_clean":"2024-03-28T15:01:33.460849+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:01:33.460849+0000","last_undegraded":"2024-03-28T15:01:33.460849+0000","last_fullsized":"2024-03-28T15:01:33.460849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:05.942532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460385+0000","last_change":"2024-03-28T15:01:31.093265+0000","last_active":"2024-03-28T15:01:33.460385+0000","last_peered":"2024-03-28T15:01:33.460385+0000","last_clean":"2024-03-28T15:01:33.460385+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:01:33.460385+0000","last_undegraded":"2024-03-28T15:01:33.460385+0000","last_fullsized":"2024-03-28T15:01:33.460385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104892+0000","last_change":"2024-03-28T15:01:32.108597+0000","last_active":"2024-03-28T15:01:33.104892+0000","last_peered":"2024-03-28T15:01:33.104892+0000","last_clean":"2024-03-28T15:01:33.104892+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:01:33.104892+0000","last_undegraded":"2024-03-28T15:01:33.104892+0000","last_fullsized":"2024-03-28T15:01:33.104892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108209+0000","last_change":"2024-03-28T15:01:31.094750+0000","last_active":"2024-03-28T15:01:33.108209+0000","last_peered":"2024-03-28T15:01:33.108209+0000","last_clean":"2024-03-28T15:01:33.108209+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:01:33.108209+0000","last_undegraded":"2024-03-28T15:01:33.108209+0000","last_fullsized":"2024-03-28T15:01:33.108209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106970+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.106970+0000","last_peered":"2024-03-28T15:01:33.106970+0000","last_clean":"2024-03-28T15:01:33.106970+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:01:33.106970+0000","last_undegraded":"2024-03-28T15:01:33.106970+0000","last_fullsized":"2024-03-28T15:01:33.106970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:24.747778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104874+0000","last_change":"2024-03-28T15:01:31.093706+0000","last_active":"2024-03-28T15:01:33.104874+0000","last_peered":"2024-03-28T15:01:33.104874+0000","last_clean":"2024-03-28T15:01:33.104874+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:01:33.104874+0000","last_undegraded":"2024-03-28T15:01:33.104874+0000","last_fullsized":"2024-03-28T15:01:33.104874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108112+0000","last_change":"2024-03-28T15:01:32.122840+0000","last_active":"2024-03-28T15:01:33.108112+0000","last_peered":"2024-03-28T15:01:33.108112+0000","last_clean":"2024-03-28T15:01:33.108112+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:01:33.108112+0000","last_undegraded":"2024-03-28T15:01:33.108112+0000","last_fullsized":"2024-03-28T15:01:33.108112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461371+0000","last_change":"2024-03-28T15:01:31.095126+0000","last_active":"2024-03-28T15:01:33.461371+0000","last_peered":"2024-03-28T15:01:33.461371+0000","last_clean":"2024-03-28T15:01:33.461371+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:01:33.461371+0000","last_undegraded":"2024-03-28T15:01:33.461371+0000","last_fullsized":"2024-03-28T15:01:33.461371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:21.669535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104885+0000","last_change":"2024-03-28T15:01:32.111925+0000","last_active":"2024-03-28T15:01:33.104885+0000","last_peered":"2024-03-28T15:01:33.104885+0000","last_clean":"2024-03-28T15:01:33.104885+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:01:33.104885+0000","last_undegraded":"2024-03-28T15:01:33.104885+0000","last_fullsized":"2024-03-28T15:01:33.104885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104964+0000","last_change":"2024-03-28T15:01:31.093197+0000","last_active":"2024-03-28T15:01:33.104964+0000","last_peered":"2024-03-28T15:01:33.104964+0000","last_clean":"2024-03-28T15:01:33.104964+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:01:33.104964+0000","last_undegraded":"2024-03-28T15:01:33.104964+0000","last_fullsized":"2024-03-28T15:01:33.104964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461153+0000","last_change":"2024-03-28T15:01:32.115432+0000","last_active":"2024-03-28T15:01:33.461153+0000","last_peered":"2024-03-28T15:01:33.461153+0000","last_clean":"2024-03-28T15:01:33.461153+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:01:33.461153+0000","last_undegraded":"2024-03-28T15:01:33.461153+0000","last_fullsized":"2024-03-28T15:01:33.461153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.125948+0000","last_change":"2024-03-28T15:01:31.088204+0000","last_active":"2024-03-28T15:01:37.125948+0000","last_peered":"2024-03-28T15:01:37.125948+0000","last_clean":"2024-03-28T15:01:37.125948+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:01:37.125948+0000","last_undegraded":"2024-03-28T15:01:37.125948+0000","last_fullsized":"2024-03-28T15:01:37.125948+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461225+0000","last_change":"2024-03-28T15:01:32.114331+0000","last_active":"2024-03-28T15:01:33.461225+0000","last_peered":"2024-03-28T15:01:33.461225+0000","last_clean":"2024-03-28T15:01:33.461225+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:01:33.461225+0000","last_undegraded":"2024-03-28T15:01:33.461225+0000","last_fullsized":"2024-03-28T15:01:33.461225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460969+0000","last_change":"2024-03-28T15:01:31.095371+0000","last_active":"2024-03-28T15:01:33.460969+0000","last_peered":"2024-03-28T15:01:33.460969+0000","last_clean":"2024-03-28T15:01:33.460969+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:01:33.460969+0000","last_undegraded":"2024-03-28T15:01:33.460969+0000","last_fullsized":"2024-03-28T15:01:33.460969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107409+0000","last_change":"2024-03-28T15:01:32.110950+0000","last_active":"2024-03-28T15:01:33.107409+0000","last_peered":"2024-03-28T15:01:33.107409+0000","last_clean":"2024-03-28T15:01:33.107409+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:01:33.107409+0000","last_undegraded":"2024-03-28T15:01:33.107409+0000","last_fullsized":"2024-03-28T15:01:33.107409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105009+0000","last_change":"2024-03-28T15:01:31.094542+0000","last_active":"2024-03-28T15:01:33.105009+0000","last_peered":"2024-03-28T15:01:33.105009+0000","last_clean":"2024-03-28T15:01:33.105009+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:01:33.105009+0000","last_undegraded":"2024-03-28T15:01:33.105009+0000","last_fullsized":"2024-03-28T15:01:33.105009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460466+0000","last_change":"2024-03-28T15:01:32.112158+0000","last_active":"2024-03-28T15:01:33.460466+0000","last_peered":"2024-03-28T15:01:33.460466+0000","last_clean":"2024-03-28T15:01:33.460466+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:01:33.460466+0000","last_undegraded":"2024-03-28T15:01:33.460466+0000","last_fullsized":"2024-03-28T15:01:33.460466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104884+0000","last_change":"2024-03-28T15:01:31.094440+0000","last_active":"2024-03-28T15:01:33.104884+0000","last_peered":"2024-03-28T15:01:33.104884+0000","last_clean":"2024-03-28T15:01:33.104884+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:01:33.104884+0000","last_undegraded":"2024-03-28T15:01:33.104884+0000","last_fullsized":"2024-03-28T15:01:33.104884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460758+0000","last_change":"2024-03-28T15:01:32.115192+0000","last_active":"2024-03-28T15:01:33.460758+0000","last_peered":"2024-03-28T15:01:33.460758+0000","last_clean":"2024-03-28T15:01:33.460758+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:01:33.460758+0000","last_undegraded":"2024-03-28T15:01:33.460758+0000","last_fullsized":"2024-03-28T15:01:33.460758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.125967+0000","last_change":"2024-03-28T15:01:31.096352+0000","last_active":"2024-03-28T15:01:37.125967+0000","last_peered":"2024-03-28T15:01:37.125967+0000","last_clean":"2024-03-28T15:01:37.125967+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:01:37.125967+0000","last_undegraded":"2024-03-28T15:01:37.125967+0000","last_fullsized":"2024-03-28T15:01:37.125967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460555+0000","last_change":"2024-03-28T15:01:32.124700+0000","last_active":"2024-03-28T15:01:33.460555+0000","last_peered":"2024-03-28T15:01:33.460555+0000","last_clean":"2024-03-28T15:01:33.460555+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:01:33.460555+0000","last_undegraded":"2024-03-28T15:01:33.460555+0000","last_fullsized":"2024-03-28T15:01:33.460555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107999+0000","last_change":"2024-03-28T15:01:32.123638+0000","last_active":"2024-03-28T15:01:33.107999+0000","last_peered":"2024-03-28T15:01:33.107999+0000","last_clean":"2024-03-28T15:01:33.107999+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:01:33.107999+0000","last_undegraded":"2024-03-28T15:01:33.107999+0000","last_fullsized":"2024-03-28T15:01:33.107999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.121539+0000","last_change":"2024-03-28T15:01:31.094281+0000","last_active":"2024-03-28T15:01:37.121539+0000","last_peered":"2024-03-28T15:01:37.121539+0000","last_clean":"2024-03-28T15:01:37.121539+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:01:37.121539+0000","last_undegraded":"2024-03-28T15:01:37.121539+0000","last_fullsized":"2024-03-28T15:01:37.121539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107988+0000","last_change":"2024-03-28T15:01:32.112496+0000","last_active":"2024-03-28T15:01:33.107988+0000","last_peered":"2024-03-28T15:01:33.107988+0000","last_clean":"2024-03-28T15:01:33.107988+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:01:33.107988+0000","last_undegraded":"2024-03-28T15:01:33.107988+0000","last_fullsized":"2024-03-28T15:01:33.107988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126177+0000","last_change":"2024-03-28T15:01:31.093773+0000","last_active":"2024-03-28T15:01:37.126177+0000","last_peered":"2024-03-28T15:01:37.126177+0000","last_clean":"2024-03-28T15:01:37.126177+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:01:37.126177+0000","last_undegraded":"2024-03-28T15:01:37.126177+0000","last_fullsized":"2024-03-28T15:01:37.126177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:54.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460976+0000","last_change":"2024-03-28T15:01:32.117275+0000","last_active":"2024-03-28T15:01:33.460976+0000","last_peered":"2024-03-28T15:01:33.460976+0000","last_clean":"2024-03-28T15:01:33.460976+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:01:33.460976+0000","last_undegraded":"2024-03-28T15:01:33.460976+0000","last_fullsized":"2024-03-28T15:01:33.460976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.475554+0000","last_change":"2024-03-28T15:01:31.096725+0000","last_active":"2024-03-28T15:01:34.475554+0000","last_peered":"2024-03-28T15:01:34.475554+0000","last_clean":"2024-03-28T15:01:34.475554+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:01:34.475554+0000","last_undegraded":"2024-03-28T15:01:34.475554+0000","last_fullsized":"2024-03-28T15:01:34.475554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460734+0000","last_change":"2024-03-28T15:01:32.120071+0000","last_active":"2024-03-28T15:01:33.460734+0000","last_peered":"2024-03-28T15:01:33.460734+0000","last_clean":"2024-03-28T15:01:33.460734+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:01:33.460734+0000","last_undegraded":"2024-03-28T15:01:33.460734+0000","last_fullsized":"2024-03-28T15:01:33.460734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126217+0000","last_change":"2024-03-28T15:01:31.095351+0000","last_active":"2024-03-28T15:01:37.126217+0000","last_peered":"2024-03-28T15:01:37.126217+0000","last_clean":"2024-03-28T15:01:37.126217+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:01:37.126217+0000","last_undegraded":"2024-03-28T15:01:37.126217+0000","last_fullsized":"2024-03-28T15:01:37.126217+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:01:40.794 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-28T15:01:40.794 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T15:01:40.794 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-28T15:01:40.794 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-28T15:01:40.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.931+0000 7f5254f8c640 1 Processor -- start 2024-03-28T15:01:40.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.931+0000 7f5254f8c640 1 -- start start 2024-03-28T15:01:40.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.932+0000 7f5254f8c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f52500c2d00 0x7f52500719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.932+0000 7f5254f8c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f525006caa0 con 0x7f5250151990 2024-03-28T15:01:40.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.932+0000 7f5254f8c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f525006cd80 con 0x7f52500b1e30 2024-03-28T15:01:40.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.932+0000 7f5254f8c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f52500c1da0 con 0x7f52500c2d00 2024-03-28T15:01:40.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.932+0000 7f524ed9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f52500c2d00 0x7f52500719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.932+0000 7f524ed9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f52500c2d00 0x7f52500719c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48448/0 (socket says 172.21.15.186:48448) 2024-03-28T15:01:40.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.932+0000 7f524ed9d640 1 -- 172.21.15.186:0/1790547397 learned_addr learned my addr 172.21.15.186:0/1790547397 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:40.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.933+0000 7f524ed9d640 1 -- 172.21.15.186:0/1790547397 >> v1:172.21.15.191:6789/0 conn(0x7f52500b1e30 legacy=0x7f52500712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.933+0000 7f524d59a640 1 -- 172.21.15.186:0/1790547397 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845539082 0 0) 0x7f525006cd80 con 0x7f52500b1e30 2024-03-28T15:01:40.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.933+0000 7f524ed9d640 1 -- 172.21.15.186:0/1790547397 >> v1:172.21.15.186:6789/0 conn(0x7f5250151990 legacy=0x7f52500c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.933+0000 7f524ed9d640 1 -- 172.21.15.186:0/1790547397 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52500c1340 con 0x7f52500c2d00 2024-03-28T15:01:40.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.933+0000 7f524ed9d640 1 --2- 172.21.15.186:0/1790547397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52500c2d00 0x7f52500719c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f524800bb10 tx=0x7f524802e280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea26f2c79d95008 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:40.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.933+0000 7f524d59a640 1 -- 172.21.15.186:0/1790547397 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524803b070 con 0x7f52500c2d00 2024-03-28T15:01:40.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.933+0000 7f524d59a640 1 -- 172.21.15.186:0/1790547397 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5248036640 con 0x7f52500c2d00 2024-03-28T15:01:40.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.934+0000 7f524d59a640 1 -- 172.21.15.186:0/1790547397 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5248036a10 con 0x7f52500c2d00 2024-03-28T15:01:40.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.934+0000 7f5254f8c640 1 -- 172.21.15.186:0/1790547397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52500c2d00 msgr2=0x7f52500719c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.934+0000 7f5254f8c640 1 --2- 172.21.15.186:0/1790547397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52500c2d00 0x7f52500719c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f524800bb10 tx=0x7f524802e280 comp rx=0 tx=0).stop 2024-03-28T15:01:40.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.934+0000 7f5254f8c640 1 -- 172.21.15.186:0/1790547397 shutdown_connections 2024-03-28T15:01:40.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.934+0000 7f5254f8c640 1 --2- 172.21.15.186:0/1790547397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52500c2d00 0x7f52500719c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.934+0000 7f5254f8c640 1 -- 172.21.15.186:0/1790547397 >> 172.21.15.186:0/1790547397 conn(0x7f52500c8a20 msgr2=0x7f52500c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:40.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.934+0000 7f5254f8c640 1 -- 172.21.15.186:0/1790547397 shutdown_connections 2024-03-28T15:01:40.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.934+0000 7f5254f8c640 1 -- 172.21.15.186:0/1790547397 wait complete. 2024-03-28T15:01:40.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.935+0000 7f5254f8c640 1 Processor -- start 2024-03-28T15:01:40.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.935+0000 7f5254f8c640 1 -- start start 2024-03-28T15:01:40.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.936+0000 7f5254f8c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52500b1e30 0x7f52500cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.936+0000 7f5254f8c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52500c2d00 0x7f52500ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.936+0000 7f5254f8c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5250151990 0x7f525014c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.936+0000 7f5254f8c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52500c2940 con 0x7f52500b1e30 2024-03-28T15:01:40.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.936+0000 7f5254f8c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f52500c1900 con 0x7f5250151990 2024-03-28T15:01:40.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.936+0000 7f5254f8c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f52500c2ac0 con 0x7f52500c2d00 2024-03-28T15:01:40.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.936+0000 7f524e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52500b1e30 0x7f52500cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.936+0000 7f524e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52500b1e30 0x7f52500cb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48456/0 (socket says 172.21.15.186:48456) 2024-03-28T15:01:40.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.936+0000 7f524e59c640 1 -- 172.21.15.186:0/1263195737 learned_addr learned my addr 172.21.15.186:0/1263195737 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.936+0000 7f524ed9d640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5250151990 0x7f525014c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.936+0000 7f524dd9b640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52500c2d00 0x7f52500ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f524e59c640 1 -- 172.21.15.186:0/1263195737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52500c2d00 msgr2=0x7f52500ca730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f524e59c640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52500c2d00 0x7f52500ca730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f524e59c640 1 -- 172.21.15.186:0/1263195737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5250151990 msgr2=0x7f525014c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f524e59c640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5250151990 0x7f525014c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f524e59c640 1 -- 172.21.15.186:0/1263195737 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52500b0160 con 0x7f52500b1e30 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f524dd9b640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52500c2d00 0x7f52500ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f524e59c640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52500b1e30 0x7f52500cb2b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f524000c7c0 tx=0x7f524000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f523effd640 1 -- 172.21.15.186:0/1263195737 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240016020 con 0x7f52500b1e30 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f523effd640 1 -- 172.21.15.186:0/1263195737 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5240006d50 con 0x7f52500b1e30 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f5254f8c640 1 -- 172.21.15.186:0/1263195737 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52500b0400 con 0x7f52500b1e30 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f523effd640 1 -- 172.21.15.186:0/1263195737 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240003bb0 con 0x7f52500b1e30 2024-03-28T15:01:40.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.937+0000 7f5254f8c640 1 -- 172.21.15.186:0/1263195737 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52500b0ca0 con 0x7f52500b1e30 2024-03-28T15:01:40.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.938+0000 7f5254f8c640 1 -- 172.21.15.186:0/1263195737 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5250069500 con 0x7f52500b1e30 2024-03-28T15:01:40.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.938+0000 7f523effd640 1 -- 172.21.15.186:0/1263195737 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f524000a7f0 con 0x7f52500b1e30 2024-03-28T15:01:40.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.939+0000 7f523effd640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f523006c530 0x7f523006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:40.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.939+0000 7f523effd640 1 -- 172.21.15.186:0/1263195737 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f524008b2f0 con 0x7f52500b1e30 2024-03-28T15:01:40.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.939+0000 7f524dd9b640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f523006c530 0x7f523006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:40.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.939+0000 7f524dd9b640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f523006c530 0x7f523006e9f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f52500cb7f0 tx=0x7f5244009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:40.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:40.943+0000 7f523effd640 1 -- 172.21.15.186:0/1263195737 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5240020020 con 0x7f52500b1e30 2024-03-28T15:01:41.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.160+0000 7f5254f8c640 1 -- 172.21.15.186:0/1263195737 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f52500b1a00 con 0x7f52500b1e30 2024-03-28T15:01:41.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.160+0000 7f523effd640 1 -- 172.21.15.186:0/1263195737 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5240059910 con 0x7f52500b1e30 2024-03-28T15:01:41.161 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:41.161 INFO:teuthology.orchestra.run.smithi186.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T15:01:41.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.162+0000 7f5254f8c640 1 -- 172.21.15.186:0/1263195737 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f523006c530 msgr2=0x7f523006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.162+0000 7f5254f8c640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f523006c530 0x7f523006e9f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f52500cb7f0 tx=0x7f5244009040 comp rx=0 tx=0).stop 2024-03-28T15:01:41.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.162+0000 7f5254f8c640 1 -- 172.21.15.186:0/1263195737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52500b1e30 msgr2=0x7f52500cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.162+0000 7f5254f8c640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52500b1e30 0x7f52500cb2b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f524000c7c0 tx=0x7f524000cc80 comp rx=0 tx=0).stop 2024-03-28T15:01:41.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.162+0000 7f5254f8c640 1 -- 172.21.15.186:0/1263195737 shutdown_connections 2024-03-28T15:01:41.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.162+0000 7f5254f8c640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f523006c530 0x7f523006e9f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.162+0000 7f5254f8c640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5250151990 0x7f525014c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.162+0000 7f5254f8c640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52500c2d00 0x7f52500ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.162+0000 7f5254f8c640 1 --2- 172.21.15.186:0/1263195737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52500b1e30 0x7f52500cb2b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.162+0000 7f5254f8c640 1 -- 172.21.15.186:0/1263195737 >> 172.21.15.186:0/1263195737 conn(0x7f52500c8a20 msgr2=0x7f525013f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:41.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.162+0000 7f5254f8c640 1 -- 172.21.15.186:0/1263195737 shutdown_connections 2024-03-28T15:01:41.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.163+0000 7f5254f8c640 1 -- 172.21.15.186:0/1263195737 wait complete. 2024-03-28T15:01:41.174 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-28T15:01:41.174 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:41.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.275+0000 7fe5e5c9a640 1 Processor -- start 2024-03-28T15:01:41.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.275+0000 7fe5e5c9a640 1 -- start start 2024-03-28T15:01:41.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5e5c9a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe5e00b1e30 0x7fe5e00712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5e5c9a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5e006caa0 con 0x7fe5e0151990 2024-03-28T15:01:41.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5e5c9a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5e006cd80 con 0x7fe5e00c2d00 2024-03-28T15:01:41.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5e5c9a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fe5e00c1da0 con 0x7fe5e00b1e30 2024-03-28T15:01:41.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5df7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe5e0151990 0x7fe5e00c2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51372/0 (socket says 172.21.15.186:51372) 2024-03-28T15:01:41.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5df7fe640 1 -- 172.21.15.186:0/1418999061 learned_addr learned my addr 172.21.15.186:0/1418999061 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:41.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 113052301 0 0) 0x7fe5e006caa0 con 0x7fe5e0151990 2024-03-28T15:01:41.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5c8003520 con 0x7fe5e0151990 2024-03-28T15:01:41.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5deffd640 1 --2- 172.21.15.186:0/1418999061 >> v2:172.21.15.191:3300/0 conn(0x7fe5e00b1e30 0x7fe5e00712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4224877057 0 0) 0x7fe5c8003520 con 0x7fe5e0151990 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 >> v2:172.21.15.191:3300/0 conn(0x7fe5e00b1e30 msgr2=0x7fe5e00712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 --2- 172.21.15.186:0/1418999061 >> v2:172.21.15.191:3300/0 conn(0x7fe5e00b1e30 0x7fe5e00712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 >> v1:172.21.15.191:6789/0 conn(0x7fe5e00c2d00 legacy=0x7fe5e00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e00c1340 con 0x7fe5e0151990 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7fe5d0001670 con 0x7fe5e0151990 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5deffd640 1 --2- 172.21.15.186:0/1418999061 >> v2:172.21.15.191:3300/0 conn(0x7fe5e00b1e30 0x7fe5e00712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 >> v1:172.21.15.186:6789/0 conn(0x7fe5e0151990 legacy=0x7fe5e00c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 --2- 172.21.15.186:0/1418999061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5c80039b0 0x7fe5c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 --2- 172.21.15.186:0/1418999061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c8004800 0x7fe5c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 --2- 172.21.15.186:0/1418999061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 0x7fe5c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e00c1da0 con 0x7fe5c80051c0 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c800c0f0 con 0x7fe5c80039b0 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c800c2f0 con 0x7fe5c8004800 2024-03-28T15:01:41.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5dffff640 1 --2- 172.21.15.186:0/1418999061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 0x7fe5c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5df7fe640 1 --2- 172.21.15.186:0/1418999061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5c80039b0 0x7fe5c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.276+0000 7fe5deffd640 1 --2- 172.21.15.186:0/1418999061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c8004800 0x7fe5c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.277+0000 7fe5dffff640 1 -- 172.21.15.186:0/1418999061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c8004800 msgr2=0x7fe5c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:41.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.277+0000 7fe5dffff640 1 --2- 172.21.15.186:0/1418999061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c8004800 0x7fe5c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.277+0000 7fe5dffff640 1 -- 172.21.15.186:0/1418999061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5c80039b0 msgr2=0x7fe5c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:41.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.277+0000 7fe5dffff640 1 --2- 172.21.15.186:0/1418999061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5c80039b0 0x7fe5c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.277+0000 7fe5dffff640 1 -- 172.21.15.186:0/1418999061 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c8003520 con 0x7fe5c80051c0 2024-03-28T15:01:41.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.277+0000 7fe5dffff640 1 -- 172.21.15.186:0/1418999061 reap_dead start 2024-03-28T15:01:41.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.277+0000 7fe5dffff640 1 --2- 172.21.15.186:0/1418999061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 0x7fe5c800b9c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8001e50 tx=0x7fe5c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79309bf5e7ddbbaa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:41.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.277+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8001a70 con 0x7fe5c80051c0 2024-03-28T15:01:41.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.277+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5d8003bb0 con 0x7fe5c80051c0 2024-03-28T15:01:41.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.278+0000 7fe5de7fc640 1 -- 172.21.15.186:0/1418999061 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d80050d0 con 0x7fe5c80051c0 2024-03-28T15:01:41.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.278+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/1418999061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 msgr2=0x7fe5c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.278+0000 7fe5e5c9a640 1 --2- 172.21.15.186:0/1418999061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 0x7fe5c800b9c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8001e50 tx=0x7fe5c8005830 comp rx=0 tx=0).stop 2024-03-28T15:01:41.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.278+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/1418999061 shutdown_connections 2024-03-28T15:01:41.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.278+0000 7fe5e5c9a640 1 --2- 172.21.15.186:0/1418999061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 0x7fe5c800b9c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.278+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/1418999061 >> 172.21.15.186:0/1418999061 conn(0x7fe5e00c8a20 msgr2=0x7fe5e00c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:41.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.278+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/1418999061 shutdown_connections 2024-03-28T15:01:41.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.278+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/1418999061 wait complete. 2024-03-28T15:01:41.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.279+0000 7fe5e5c9a640 1 Processor -- start 2024-03-28T15:01:41.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.279+0000 7fe5e5c9a640 1 -- start start 2024-03-28T15:01:41.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5e5c9a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 0x7fe5e014f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5e5c9a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5e01f46e0 0x7fe5e01f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5e5c9a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5e006d6d0 0x7fe5e01f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5e5c9a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e0073dd0 con 0x7fe5c80051c0 2024-03-28T15:01:41.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5e5c9a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e00c1900 con 0x7fe5e01f46e0 2024-03-28T15:01:41.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5e5c9a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fe5e0073f50 con 0x7fe5e006d6d0 2024-03-28T15:01:41.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5df7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 0x7fe5e014f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5df7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 0x7fe5e014f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48482/0 (socket says 172.21.15.186:48482) 2024-03-28T15:01:41.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5df7fe640 1 -- 172.21.15.186:0/2464715691 learned_addr learned my addr 172.21.15.186:0/2464715691 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:41.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5dffff640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5e006d6d0 0x7fe5e01f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5df7fe640 1 -- 172.21.15.186:0/2464715691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5e006d6d0 msgr2=0x7fe5e01f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5deffd640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5e01f46e0 0x7fe5e01f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5df7fe640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5e006d6d0 0x7fe5e01f32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5df7fe640 1 -- 172.21.15.186:0/2464715691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5e01f46e0 msgr2=0x7fe5e01f2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5df7fe640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5e01f46e0 0x7fe5e01f2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5df7fe640 1 -- 172.21.15.186:0/2464715691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e01f1d00 con 0x7fe5c80051c0 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.280+0000 7fe5deffd640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5e01f46e0 0x7fe5e01f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.281+0000 7fe5df7fe640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 0x7fe5e014f870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe5d00072c0 tx=0x7fe5d0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.281+0000 7fe5c7fff640 1 -- 172.21.15.186:0/2464715691 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d001a070 con 0x7fe5c80051c0 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.281+0000 7fe5c7fff640 1 -- 172.21.15.186:0/2464715691 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5d000f960 con 0x7fe5c80051c0 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.281+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/2464715691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e01f1fe0 con 0x7fe5c80051c0 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.281+0000 7fe5c7fff640 1 -- 172.21.15.186:0/2464715691 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d000fd50 con 0x7fe5c80051c0 2024-03-28T15:01:41.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.281+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/2464715691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e01f25a0 con 0x7fe5c80051c0 2024-03-28T15:01:41.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.282+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/2464715691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5e0069500 con 0x7fe5c80051c0 2024-03-28T15:01:41.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.282+0000 7fe5c7fff640 1 -- 172.21.15.186:0/2464715691 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe5d001d070 con 0x7fe5c80051c0 2024-03-28T15:01:41.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.282+0000 7fe5c7fff640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe5cc06c560 0x7fe5cc06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.283+0000 7fe5c7fff640 1 -- 172.21.15.186:0/2464715691 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe5d008b160 con 0x7fe5c80051c0 2024-03-28T15:01:41.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.283+0000 7fe5deffd640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe5cc06c560 0x7fe5cc06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.283+0000 7fe5deffd640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe5cc06c560 0x7fe5cc06ea20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4006fa0 tx=0x7fe5d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:41.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.287+0000 7fe5c7fff640 1 -- 172.21.15.186:0/2464715691 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fe5d0059780 con 0x7fe5c80051c0 2024-03-28T15:01:41.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.465+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/2464715691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5e00b1a00 con 0x7fe5c80051c0 2024-03-28T15:01:41.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.466+0000 7fe5c7fff640 1 -- 172.21.15.186:0/2464715691 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7fe5d0059080 con 0x7fe5c80051c0 2024-03-28T15:01:41.467 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:41.467 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:33.099254+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:41.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/2464715691 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe5cc06c560 msgr2=0x7fe5cc06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe5cc06c560 0x7fe5cc06ea20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4006fa0 tx=0x7fe5d4008040 comp rx=0 tx=0).stop 2024-03-28T15:01:41.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/2464715691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 msgr2=0x7fe5e014f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 0x7fe5e014f870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe5d00072c0 tx=0x7fe5d0007780 comp rx=0 tx=0).stop 2024-03-28T15:01:41.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/2464715691 shutdown_connections 2024-03-28T15:01:41.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe5cc06c560 0x7fe5cc06ea20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5e006d6d0 0x7fe5e01f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5e01f46e0 0x7fe5e01f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 --2- 172.21.15.186:0/2464715691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5c80051c0 0x7fe5e014f870 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/2464715691 >> 172.21.15.186:0/2464715691 conn(0x7fe5e00c8a20 msgr2=0x7fe5e00c7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:41.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/2464715691 shutdown_connections 2024-03-28T15:01:41.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.468+0000 7fe5e5c9a640 1 -- 172.21.15.186:0/2464715691 wait complete. 2024-03-28T15:01:41.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:01:18.661274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:01:29.512445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:01:30.388340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:01:41.481 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:01:41.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.583+0000 7f8c9230f640 1 Processor -- start 2024-03-28T15:01:41.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.584+0000 7f8c9230f640 1 -- start start 2024-03-28T15:01:41.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.584+0000 7f8c9230f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c8c154cd0 0x7f8c8c0c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.584+0000 7f8c9230f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8c8c072dd0 0x7f8c8c067280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.584+0000 7f8c9230f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c8c06ba40 con 0x7f8c8c0c79a0 2024-03-28T15:01:41.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.584+0000 7f8c9230f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f8c8c06c550 con 0x7f8c8c072dd0 2024-03-28T15:01:41.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.584+0000 7f8c9230f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8c8c06bce0 con 0x7f8c8c154cd0 2024-03-28T15:01:41.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.584+0000 7f8c908ac640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8c8c0c79a0 0x7f8c8c0734b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51388/0 (socket says 172.21.15.186:51388) 2024-03-28T15:01:41.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.584+0000 7f8c908ac640 1 -- 172.21.15.186:0/3349514028 learned_addr learned my addr 172.21.15.186:0/3349514028 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:41.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8b7fe640 1 --2- 172.21.15.186:0/3349514028 >> v2:172.21.15.191:3300/0 conn(0x7f8c8c072dd0 0x7f8c8c067280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.589 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8bfff640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c8c154cd0 0x7f8c8c0c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 26576005 0 0) 0x7f8c8c06ba40 con 0x7f8c8c0c79a0 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c74003520 con 0x7f8c8c0c79a0 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2384648274 0 0) 0x7f8c74003520 con 0x7f8c8c0c79a0 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c8c154cd0 msgr2=0x7f8c8c0c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c8c154cd0 0x7f8c8c0c93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 >> v2:172.21.15.191:3300/0 conn(0x7f8c8c072dd0 msgr2=0x7f8c8c067280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 --2- 172.21.15.186:0/3349514028 >> v2:172.21.15.191:3300/0 conn(0x7f8c8c072dd0 0x7f8c8c067280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8c06bfc0 con 0x7f8c8c0c79a0 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f8c84001a10 con 0x7f8c8c0c79a0 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 >> v1:172.21.15.186:6789/0 conn(0x7f8c8c0c79a0 legacy=0x7f8c8c0734b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c740039b0 0x7f8c74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c8c154cd0 0x7f8c74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c74004d40 0x7f8c740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c06c550 con 0x7f8c74004d40 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c06bce0 con 0x7f8c740039b0 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8c7400b9c0 con 0x7f8c8c154cd0 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c908ac640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c74004d40 0x7f8c740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8b7fe640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c8c154cd0 0x7f8c74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c8bfff640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c740039b0 0x7f8c74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c908ac640 1 -- 172.21.15.186:0/3349514028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c8c154cd0 msgr2=0x7f8c74004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c908ac640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c8c154cd0 0x7f8c74004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c908ac640 1 -- 172.21.15.186:0/3349514028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c740039b0 msgr2=0x7f8c74003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c908ac640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c740039b0 0x7f8c74003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.585+0000 7f8c908ac640 1 -- 172.21.15.186:0/3349514028 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c74003520 con 0x7f8c74004d40 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.586+0000 7f8c908ac640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c74004d40 0x7f8c740051c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400a890 tx=0x7f8c74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec17367f9f9cc3a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:41.591 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.586+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8400a4e0 con 0x7f8c74004d40 2024-03-28T15:01:41.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.586+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c840072f0 con 0x7f8c74004d40 2024-03-28T15:01:41.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.586+0000 7f8c8affd640 1 -- 172.21.15.186:0/3349514028 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c840076e0 con 0x7f8c74004d40 2024-03-28T15:01:41.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.587+0000 7f8c9230f640 1 -- 172.21.15.186:0/3349514028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c74004d40 msgr2=0x7f8c740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.587+0000 7f8c9230f640 1 --2- 172.21.15.186:0/3349514028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c74004d40 0x7f8c740051c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400a890 tx=0x7f8c74005830 comp rx=0 tx=0).stop 2024-03-28T15:01:41.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.587+0000 7f8c908ac640 1 -- 172.21.15.186:0/3349514028 reap_dead start 2024-03-28T15:01:41.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.587+0000 7f8c9230f640 1 -- 172.21.15.186:0/3349514028 shutdown_connections 2024-03-28T15:01:41.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.587+0000 7f8c9230f640 1 -- 172.21.15.186:0/3349514028 >> 172.21.15.186:0/3349514028 conn(0x7f8c8c0cdba0 msgr2=0x7f8c8c0c9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:41.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.587+0000 7f8c9230f640 1 -- 172.21.15.186:0/3349514028 shutdown_connections 2024-03-28T15:01:41.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.587+0000 7f8c9230f640 1 -- 172.21.15.186:0/3349514028 wait complete. 2024-03-28T15:01:41.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.588+0000 7f8c9230f640 1 Processor -- start 2024-03-28T15:01:41.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.588+0000 7f8c9230f640 1 -- start start 2024-03-28T15:01:41.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c9230f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c74004d40 0x7f8c8c147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c9230f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c8c147f60 0x7f8c8c144850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c9230f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c8c145410 0x7f8c8c144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c9230f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c1550b0 con 0x7f8c8c145410 2024-03-28T15:01:41.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c9230f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c06ca80 con 0x7f8c8c147f60 2024-03-28T15:01:41.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c9230f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8c8c155230 con 0x7f8c74004d40 2024-03-28T15:01:41.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c908ac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c8c145410 0x7f8c8c144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c908ac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c8c145410 0x7f8c8c144d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48498/0 (socket says 172.21.15.186:48498) 2024-03-28T15:01:41.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c908ac640 1 -- 172.21.15.186:0/1744858172 learned_addr learned my addr 172.21.15.186:0/1744858172 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:41.595 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c908ac640 1 -- 172.21.15.186:0/1744858172 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c74004d40 msgr2=0x7f8c8c147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c908ac640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c74004d40 0x7f8c8c147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c8b7fe640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c8c147f60 0x7f8c8c144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c908ac640 1 -- 172.21.15.186:0/1744858172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c8c147f60 msgr2=0x7f8c8c144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c908ac640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c8c147f60 0x7f8c8c144850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c908ac640 1 -- 172.21.15.186:0/1744858172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8c15ec70 con 0x7f8c8c145410 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c908ac640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c8c145410 0x7f8c8c144d90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400a9b0 tx=0x7f8c8c0c89b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.589+0000 7f8c88ff9640 1 -- 172.21.15.186:0/1744858172 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8400a4e0 con 0x7f8c8c145410 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.590+0000 7f8c88ff9640 1 -- 172.21.15.186:0/1744858172 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c840017d0 con 0x7f8c8c145410 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.590+0000 7f8c9230f640 1 -- 172.21.15.186:0/1744858172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c8c1459e0 con 0x7f8c8c145410 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.590+0000 7f8c88ff9640 1 -- 172.21.15.186:0/1744858172 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c84004b60 con 0x7f8c8c145410 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.590+0000 7f8c9230f640 1 -- 172.21.15.186:0/1744858172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c8c15e990 con 0x7f8c8c145410 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.590+0000 7f8c9230f640 1 -- 172.21.15.186:0/1744858172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c8c0686d0 con 0x7f8c8c145410 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.591+0000 7f8c88ff9640 1 -- 172.21.15.186:0/1744858172 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8c8403c070 con 0x7f8c8c145410 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.591+0000 7f8c88ff9640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8c7806c530 0x7f8c7806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.591+0000 7f8c8bfff640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8c7806c530 0x7f8c7806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.591+0000 7f8c88ff9640 1 -- 172.21.15.186:0/1744858172 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8c840acc30 con 0x7f8c8c145410 2024-03-28T15:01:41.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.592+0000 7f8c8bfff640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8c7806c530 0x7f8c7806e9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009a40 tx=0x7f8c7c009320 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:41.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.595+0000 7f8c88ff9640 1 -- 172.21.15.186:0/1744858172 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8c8407b1c0 con 0x7f8c8c145410 2024-03-28T15:01:41.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.772+0000 7f8c9230f640 1 -- 172.21.15.186:0/1744858172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8c8c1f0230 con 0x7f8c8c145410 2024-03-28T15:01:41.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.772+0000 7f8c88ff9640 1 -- 172.21.15.186:0/1744858172 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f8c8407aac0 con 0x7f8c8c145410 2024-03-28T15:01:41.774 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 1 2024-03-28T15:01:41.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 -- 172.21.15.186:0/1744858172 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8c7806c530 msgr2=0x7f8c7806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8c7806c530 0x7f8c7806e9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009a40 tx=0x7f8c7c009320 comp rx=0 tx=0).stop 2024-03-28T15:01:41.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 -- 172.21.15.186:0/1744858172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c8c145410 msgr2=0x7f8c8c144d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c8c145410 0x7f8c8c144d90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400a9b0 tx=0x7f8c8c0c89b0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 -- 172.21.15.186:0/1744858172 shutdown_connections 2024-03-28T15:01:41.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8c7806c530 0x7f8c7806e9f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c8c145410 0x7f8c8c144d90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c8c147f60 0x7f8c8c144850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 --2- 172.21.15.186:0/1744858172 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c74004d40 0x7f8c8c147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 -- 172.21.15.186:0/1744858172 >> 172.21.15.186:0/1744858172 conn(0x7f8c8c0cdba0 msgr2=0x7f8c8c0cb510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:41.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 -- 172.21.15.186:0/1744858172 shutdown_connections 2024-03-28T15:01:41.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.774+0000 7f8c9230f640 1 -- 172.21.15.186:0/1744858172 wait complete. 2024-03-28T15:01:41.787 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:01:41.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.889+0000 7f5ce8e0e640 1 Processor -- start 2024-03-28T15:01:41.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.889+0000 7f5ce8e0e640 1 -- start start 2024-03-28T15:01:41.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce8e0e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5ce415e620 0x7f5ce406b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce8e0e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce4070dd0 0x7f5ce4165240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce8e0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce406f710 con 0x7f5ce4157fb0 2024-03-28T15:01:41.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce8e0e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f5ce415ee00 con 0x7f5ce415e620 2024-03-28T15:01:41.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce8e0e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce40b3560 con 0x7f5ce4070dd0 2024-03-28T15:01:41.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce259c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f5ce4157fb0 0x7f5ce415df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51404/0 (socket says 172.21.15.186:51404) 2024-03-28T15:01:41.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce259c640 1 -- 172.21.15.186:0/2467065024 learned_addr learned my addr 172.21.15.186:0/2467065024 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:41.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5cd9d9b640 1 --2- 172.21.15.186:0/2467065024 >> v2:172.21.15.191:3300/0 conn(0x7f5ce415e620 0x7f5ce406b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1913670720 0 0) 0x7f5ce406f710 con 0x7f5ce4157fb0 2024-03-28T15:01:41.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce2d9d640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce4070dd0 0x7f5ce4165240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cd0003520 con 0x7f5ce4157fb0 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3370790572 0 0) 0x7f5cd0003520 con 0x7f5ce4157fb0 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce4070dd0 msgr2=0x7f5ce4165240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce4070dd0 0x7f5ce4165240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 >> v2:172.21.15.191:3300/0 conn(0x7f5ce415e620 msgr2=0x7f5ce406b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 --2- 172.21.15.186:0/2467065024 >> v2:172.21.15.191:3300/0 conn(0x7f5ce415e620 0x7f5ce406b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce415eb20 con 0x7f5ce4157fb0 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f5cd4001690 con 0x7f5ce4157fb0 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 >> v1:172.21.15.186:6789/0 conn(0x7f5ce4157fb0 legacy=0x7f5ce415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cd00039b0 0x7f5cd0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cd0004800 0x7f5cd0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce4070dd0 0x7f5cd00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce415ee00 con 0x7f5cd00039b0 2024-03-28T15:01:41.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce40b3560 con 0x7f5cd0004800 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd000b9c0 con 0x7f5ce4070dd0 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5ce259c640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cd00039b0 0x7f5cd0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.890+0000 7f5cd9d9b640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cd0004800 0x7f5cd0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.891+0000 7f5ce2d9d640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce4070dd0 0x7f5cd00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.891+0000 7f5ce259c640 1 -- 172.21.15.186:0/2467065024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce4070dd0 msgr2=0x7f5cd00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.891+0000 7f5ce259c640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce4070dd0 0x7f5cd00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.891+0000 7f5ce259c640 1 -- 172.21.15.186:0/2467065024 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cd0004800 msgr2=0x7f5cd0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.891+0000 7f5ce259c640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cd0004800 0x7f5cd0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.891+0000 7f5ce259c640 1 -- 172.21.15.186:0/2467065024 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd0003520 con 0x7f5cd00039b0 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.891+0000 7f5ce259c640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cd00039b0 0x7f5cd0003dd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4007fd0 tx=0x7f5cd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa1a187e863b378 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.892+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4007d50 con 0x7f5cd00039b0 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.892+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cd4030070 con 0x7f5cd00039b0 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.892+0000 7f5ce1d9b640 1 -- 172.21.15.186:0/2467065024 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4005d30 con 0x7f5cd00039b0 2024-03-28T15:01:41.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.892+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/2467065024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cd00039b0 msgr2=0x7f5cd0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.892+0000 7f5ce8e0e640 1 --2- 172.21.15.186:0/2467065024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cd00039b0 0x7f5cd0003dd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4007fd0 tx=0x7f5cd0005830 comp rx=0 tx=0).stop 2024-03-28T15:01:41.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.892+0000 7f5ce2d9d640 1 -- 172.21.15.186:0/2467065024 reap_dead start 2024-03-28T15:01:41.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.892+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/2467065024 shutdown_connections 2024-03-28T15:01:41.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.892+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/2467065024 >> 172.21.15.186:0/2467065024 conn(0x7f5ce4073610 msgr2=0x7f5ce406d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:41.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.892+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/2467065024 shutdown_connections 2024-03-28T15:01:41.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.892+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/2467065024 wait complete. 2024-03-28T15:01:41.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce8e0e640 1 Processor -- start 2024-03-28T15:01:41.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce8e0e640 1 -- start start 2024-03-28T15:01:41.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce8e0e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cd00039b0 0x7f5ce40cc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce8e0e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce415dfd0 0x7f5ce414f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce8e0e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4151200 0x7f5ce4151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce8e0e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4070820 con 0x7f5ce4151200 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce8e0e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce406f9b0 con 0x7f5cd00039b0 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce8e0e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce4158390 con 0x7f5ce415dfd0 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4151200 0x7f5ce4151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4151200 0x7f5ce4151680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48516/0 (socket says 172.21.15.186:48516) 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce2d9d640 1 -- 172.21.15.186:0/1770400001 learned_addr learned my addr 172.21.15.186:0/1770400001 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5cd9d9b640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce415dfd0 0x7f5ce414f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.894+0000 7f5ce259c640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cd00039b0 0x7f5ce40cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.895+0000 7f5ce2d9d640 1 -- 172.21.15.186:0/1770400001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce415dfd0 msgr2=0x7f5ce414f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.895+0000 7f5ce2d9d640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce415dfd0 0x7f5ce414f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.895+0000 7f5ce2d9d640 1 -- 172.21.15.186:0/1770400001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cd00039b0 msgr2=0x7f5ce40cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.895+0000 7f5ce2d9d640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cd00039b0 0x7f5ce40cc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:41.900 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.895+0000 7f5ce2d9d640 1 -- 172.21.15.186:0/1770400001 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce414ec10 con 0x7f5ce4151200 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.895+0000 7f5ce2d9d640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4151200 0x7f5ce4151680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc011b40 tx=0x7f5cdc011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.895+0000 7f5cdb7fe640 1 -- 172.21.15.186:0/1770400001 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc00fca0 con 0x7f5ce4151200 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.895+0000 7f5cdb7fe640 1 -- 172.21.15.186:0/1770400001 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cdc00fe40 con 0x7f5ce4151200 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.895+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/1770400001 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce4151bc0 con 0x7f5ce4151200 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.895+0000 7f5cdb7fe640 1 -- 172.21.15.186:0/1770400001 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc015670 con 0x7f5ce4151200 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.895+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/1770400001 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce40cd360 con 0x7f5ce4151200 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.896+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/1770400001 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce40681e0 con 0x7f5ce4151200 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.896+0000 7f5cdb7fe640 1 -- 172.21.15.186:0/1770400001 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5cdc015810 con 0x7f5ce4151200 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.897+0000 7f5cdb7fe640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5cc406c530 0x7f5cc406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.897+0000 7f5cdb7fe640 1 -- 172.21.15.186:0/1770400001 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5cdc08c2c0 con 0x7f5ce4151200 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.897+0000 7f5ce259c640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5cc406c530 0x7f5cc406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:41.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.897+0000 7f5ce259c640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5cc406c530 0x7f5cc406e9f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4034000 tx=0x7f5cd4007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:41.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:41.901+0000 7f5cdb7fe640 1 -- 172.21.15.186:0/1770400001 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5ce4151bc0 con 0x7f5ce4151200 2024-03-28T15:01:42.078 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.077+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/1770400001 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5ce4069570 con 0x7f5ce4151200 2024-03-28T15:01:42.079 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.077+0000 7f5cdb7fe640 1 -- 172.21.15.186:0/1770400001 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f5ce414ec10 con 0x7f5ce4151200 2024-03-28T15:01:42.079 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 32 2024-03-28T15:01:42.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/1770400001 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5cc406c530 msgr2=0x7f5cc406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5cc406c530 0x7f5cc406e9f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4034000 tx=0x7f5cd4007000 comp rx=0 tx=0).stop 2024-03-28T15:01:42.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/1770400001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4151200 msgr2=0x7f5ce4151680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4151200 0x7f5ce4151680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc011b40 tx=0x7f5cdc011f40 comp rx=0 tx=0).stop 2024-03-28T15:01:42.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/1770400001 shutdown_connections 2024-03-28T15:01:42.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5cc406c530 0x7f5cc406e9f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ce4151200 0x7f5ce4151680 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ce415dfd0 0x7f5ce414f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 --2- 172.21.15.186:0/1770400001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cd00039b0 0x7f5ce40cc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/1770400001 >> 172.21.15.186:0/1770400001 conn(0x7f5ce4073610 msgr2=0x7f5ce41e3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:42.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/1770400001 shutdown_connections 2024-03-28T15:01:42.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.079+0000 7f5ce8e0e640 1 -- 172.21.15.186:0/1770400001 wait complete. 2024-03-28T15:01:42.094 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:01:42.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.203+0000 7fbb2ee8f640 1 Processor -- start 2024-03-28T15:01:42.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.203+0000 7fbb2ee8f640 1 -- start start 2024-03-28T15:01:42.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.203+0000 7fbb2ee8f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbb280c8a20 0x7fbb2806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.203+0000 7fbb2ee8f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb280cb2b0 con 0x7fbb280b3180 2024-03-28T15:01:42.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.203+0000 7fbb2ee8f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb2806b500 con 0x7fbb2815b440 2024-03-28T15:01:42.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.203+0000 7fbb2ee8f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fbb280713a0 con 0x7fbb280c8a20 2024-03-28T15:01:42.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.204+0000 7fbb2d42c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbb280c8a20 0x7fbb2806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.204+0000 7fbb2d42c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbb280c8a20 0x7fbb2806cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48528/0 (socket says 172.21.15.186:48528) 2024-03-28T15:01:42.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.204+0000 7fbb2cc2b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fbb280b3180 0x7fbb28071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51412/0 (socket says 172.21.15.186:51412) 2024-03-28T15:01:42.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.204+0000 7fbb2d42c640 1 -- 172.21.15.186:0/1583345905 learned_addr learned my addr 172.21.15.186:0/1583345905 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:42.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.204+0000 7fbb2d42c640 1 -- 172.21.15.186:0/1583345905 >> v1:172.21.15.191:6789/0 conn(0x7fbb2815b440 legacy=0x7fbb2806e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.204+0000 7fbb2d42c640 1 -- 172.21.15.186:0/1583345905 >> v1:172.21.15.186:6789/0 conn(0x7fbb280b3180 legacy=0x7fbb28071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.204+0000 7fbb2d42c640 1 -- 172.21.15.186:0/1583345905 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2806b7e0 con 0x7fbb280c8a20 2024-03-28T15:01:42.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.205+0000 7fbb2d42c640 1 --2- 172.21.15.186:0/1583345905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb280c8a20 0x7fbb2806cff0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbb28071520 tx=0x7fbb2002c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7816f44275a26626 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:42.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.205+0000 7fbb277fe640 1 -- 172.21.15.186:0/1583345905 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2002cc10 con 0x7fbb280c8a20 2024-03-28T15:01:42.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.205+0000 7fbb277fe640 1 -- 172.21.15.186:0/1583345905 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb2002cdb0 con 0x7fbb280c8a20 2024-03-28T15:01:42.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.205+0000 7fbb277fe640 1 -- 172.21.15.186:0/1583345905 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20034610 con 0x7fbb280c8a20 2024-03-28T15:01:42.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.205+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1583345905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb280c8a20 msgr2=0x7fbb2806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.205+0000 7fbb2ee8f640 1 --2- 172.21.15.186:0/1583345905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb280c8a20 0x7fbb2806cff0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbb28071520 tx=0x7fbb2002c070 comp rx=0 tx=0).stop 2024-03-28T15:01:42.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.205+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1583345905 shutdown_connections 2024-03-28T15:01:42.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.205+0000 7fbb2ee8f640 1 --2- 172.21.15.186:0/1583345905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb280c8a20 0x7fbb2806cff0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.205+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1583345905 >> 172.21.15.186:0/1583345905 conn(0x7fbb2806fa00 msgr2=0x7fbb28073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:42.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.205+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1583345905 shutdown_connections 2024-03-28T15:01:42.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.205+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1583345905 wait complete. 2024-03-28T15:01:42.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.206+0000 7fbb2ee8f640 1 Processor -- start 2024-03-28T15:01:42.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.207+0000 7fbb2ee8f640 1 -- start start 2024-03-28T15:01:42.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.207+0000 7fbb2ee8f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb280b3180 0x7fbb281f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.207+0000 7fbb2ee8f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb280c8a20 0x7fbb2815a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.207+0000 7fbb2ee8f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb28156f60 0x7fbb280cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.207+0000 7fbb2ee8f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2815b8d0 con 0x7fbb280b3180 2024-03-28T15:01:42.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.207+0000 7fbb2ee8f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbb280b3560 con 0x7fbb280c8a20 2024-03-28T15:01:42.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.207+0000 7fbb2ee8f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fbb2806ef20 con 0x7fbb28156f60 2024-03-28T15:01:42.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.207+0000 7fbb2cc2b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb280b3180 0x7fbb281f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.207+0000 7fbb2cc2b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb280b3180 0x7fbb281f0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48530/0 (socket says 172.21.15.186:48530) 2024-03-28T15:01:42.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.207+0000 7fbb2cc2b640 1 -- 172.21.15.186:0/1870119577 learned_addr learned my addr 172.21.15.186:0/1870119577 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:42.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb27fff640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb280c8a20 0x7fbb2815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb2d42c640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb28156f60 0x7fbb280cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb2cc2b640 1 -- 172.21.15.186:0/1870119577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb28156f60 msgr2=0x7fbb280cbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb2cc2b640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb28156f60 0x7fbb280cbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb2cc2b640 1 -- 172.21.15.186:0/1870119577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb280c8a20 msgr2=0x7fbb2815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb2cc2b640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb280c8a20 0x7fbb2815a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb2cc2b640 1 -- 172.21.15.186:0/1870119577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb281f3a80 con 0x7fbb280b3180 2024-03-28T15:01:42.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb2cc2b640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb280b3180 0x7fbb281f0ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c00c500 tx=0x7fbb1c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:42.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb257fa640 1 -- 172.21.15.186:0/1870119577 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c018070 con 0x7fbb280b3180 2024-03-28T15:01:42.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb257fa640 1 -- 172.21.15.186:0/1870119577 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb1c007500 con 0x7fbb280b3180 2024-03-28T15:01:42.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1870119577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb281f3d60 con 0x7fbb280b3180 2024-03-28T15:01:42.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb257fa640 1 -- 172.21.15.186:0/1870119577 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c0049e0 con 0x7fbb280b3180 2024-03-28T15:01:42.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.208+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1870119577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb281f4320 con 0x7fbb280b3180 2024-03-28T15:01:42.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.209+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1870119577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb280686d0 con 0x7fbb280b3180 2024-03-28T15:01:42.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.210+0000 7fbb257fa640 1 -- 172.21.15.186:0/1870119577 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbb1c01b070 con 0x7fbb280b3180 2024-03-28T15:01:42.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.210+0000 7fbb257fa640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbb0c06c530 0x7fbb0c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.210+0000 7fbb27fff640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbb0c06c530 0x7fbb0c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.210+0000 7fbb257fa640 1 -- 172.21.15.186:0/1870119577 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbb1c08a990 con 0x7fbb280b3180 2024-03-28T15:01:42.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.210+0000 7fbb27fff640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbb0c06c530 0x7fbb0c06e9f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb2814bcc0 tx=0x7fbb14009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:42.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.214+0000 7fbb257fa640 1 -- 172.21.15.186:0/1870119577 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fbb1c0589f0 con 0x7fbb280b3180 2024-03-28T15:01:42.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.388+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1870119577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbb280ce600 con 0x7fbb280b3180 2024-03-28T15:01:42.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.389+0000 7fbb257fa640 1 -- 172.21.15.186:0/1870119577 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fbb1c05ccf0 con 0x7fbb280b3180 2024-03-28T15:01:42.391 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 64 2024-03-28T15:01:42.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1870119577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbb0c06c530 msgr2=0x7fbb0c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbb0c06c530 0x7fbb0c06e9f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb2814bcc0 tx=0x7fbb14009040 comp rx=0 tx=0).stop 2024-03-28T15:01:42.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1870119577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb280b3180 msgr2=0x7fbb281f0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb280b3180 0x7fbb281f0ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c00c500 tx=0x7fbb1c00c9c0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1870119577 shutdown_connections 2024-03-28T15:01:42.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbb0c06c530 0x7fbb0c06e9f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb28156f60 0x7fbb280cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb280c8a20 0x7fbb2815a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 --2- 172.21.15.186:0/1870119577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb280b3180 0x7fbb281f0ef0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1870119577 >> 172.21.15.186:0/1870119577 conn(0x7fbb2806fa00 msgr2=0x7fbb28071830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:42.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1870119577 shutdown_connections 2024-03-28T15:01:42.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.391+0000 7fbb2ee8f640 1 -- 172.21.15.186:0/1870119577 wait complete. 2024-03-28T15:01:42.404 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:01:42.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.504+0000 7f1ccb7f7640 1 Processor -- start 2024-03-28T15:01:42.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.504+0000 7f1ccb7f7640 1 -- start start 2024-03-28T15:01:42.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.505+0000 7f1ccb7f7640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1cc406d670 0x7f1cc406f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.505+0000 7f1ccb7f7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cc406f770 con 0x7f1cc406fb90 2024-03-28T15:01:42.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.505+0000 7f1ccb7f7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cc4159500 con 0x7f1cc406da50 2024-03-28T15:01:42.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.505+0000 7f1ccb7f7640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f1cc40c91a0 con 0x7f1cc406d670 2024-03-28T15:01:42.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.505+0000 7f1cc8d92640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f1cc406fb90 0x7f1cc40c8a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51426/0 (socket says 172.21.15.186:51426) 2024-03-28T15:01:42.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.505+0000 7f1cc8d92640 1 -- 172.21.15.186:0/3158783058 learned_addr learned my addr 172.21.15.186:0/3158783058 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:42.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.505+0000 7f1cc9593640 1 --2- 172.21.15.186:0/3158783058 >> v2:172.21.15.191:3300/0 conn(0x7f1cc406d670 0x7f1cc406f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.506+0000 7f1cc3fff640 1 -- 172.21.15.186:0/3158783058 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1098833045 0 0) 0x7f1cc406f770 con 0x7f1cc406fb90 2024-03-28T15:01:42.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.506+0000 7f1cc3fff640 1 -- 172.21.15.186:0/3158783058 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cac003520 con 0x7f1cc406fb90 2024-03-28T15:01:42.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.506+0000 7f1cc3fff640 1 -- 172.21.15.186:0/3158783058 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2925568812 0 0) 0x7f1cc4159500 con 0x7f1cc406da50 2024-03-28T15:01:42.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.506+0000 7f1cc3fff640 1 -- 172.21.15.186:0/3158783058 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cc406f770 con 0x7f1cc406da50 2024-03-28T15:01:42.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.506+0000 7f1cc9593640 1 -- 172.21.15.186:0/3158783058 >> v1:172.21.15.191:6789/0 conn(0x7f1cc406da50 legacy=0x7f1cc4162f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.506+0000 7f1cc9593640 1 -- 172.21.15.186:0/3158783058 >> v1:172.21.15.186:6789/0 conn(0x7f1cc406fb90 legacy=0x7f1cc40c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.506+0000 7f1cc9593640 1 -- 172.21.15.186:0/3158783058 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc4159220 con 0x7f1cc406d670 2024-03-28T15:01:42.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.507+0000 7f1cc9593640 1 --2- 172.21.15.186:0/3158783058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1cc406d670 0x7f1cc406f0a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009d90 tx=0x7f1cb402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feca19d311a605e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.507+0000 7f1cc3fff640 1 -- 172.21.15.186:0/3158783058 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4039070 con 0x7f1cc406d670 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.507+0000 7f1cc3fff640 1 -- 172.21.15.186:0/3158783058 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cb402cc60 con 0x7f1cc406d670 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.507+0000 7f1cc3fff640 1 -- 172.21.15.186:0/3158783058 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4034660 con 0x7f1cc406d670 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.507+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/3158783058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1cc406d670 msgr2=0x7f1cc406f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.507+0000 7f1ccb7f7640 1 --2- 172.21.15.186:0/3158783058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1cc406d670 0x7f1cc406f0a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009d90 tx=0x7f1cb402beb0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.507+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/3158783058 shutdown_connections 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.507+0000 7f1ccb7f7640 1 --2- 172.21.15.186:0/3158783058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1cc406d670 0x7f1cc406f0a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.507+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/3158783058 >> 172.21.15.186:0/3158783058 conn(0x7f1cc4067670 msgr2=0x7f1cc4071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.507+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/3158783058 shutdown_connections 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.507+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/3158783058 wait complete. 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.508+0000 7f1ccb7f7640 1 Processor -- start 2024-03-28T15:01:42.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.508+0000 7f1ccb7f7640 1 -- start start 2024-03-28T15:01:42.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1ccb7f7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1cc406d670 0x7f1cc414ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1ccb7f7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc406da50 0x7f1cc414fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1ccb7f7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1cc406fb90 0x7f1cc4156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1ccb7f7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc40c34c0 con 0x7f1cc406da50 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1ccb7f7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc40c4c60 con 0x7f1cc406fb90 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1ccb7f7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f1cc40c50f0 con 0x7f1cc406d670 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1cc8d92640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc406da50 0x7f1cc414fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1cc8d92640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc406da50 0x7f1cc414fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48546/0 (socket says 172.21.15.186:48546) 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1cc8d92640 1 -- 172.21.15.186:0/2613570201 learned_addr learned my addr 172.21.15.186:0/2613570201 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1cc9593640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1cc406d670 0x7f1cc414ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1cc9d94640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1cc406fb90 0x7f1cc4156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1cc8d92640 1 -- 172.21.15.186:0/2613570201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1cc406d670 msgr2=0x7f1cc414ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1cc8d92640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1cc406d670 0x7f1cc414ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1cc8d92640 1 -- 172.21.15.186:0/2613570201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1cc406fb90 msgr2=0x7f1cc4156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1cc8d92640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1cc406fb90 0x7f1cc4156470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1cc8d92640 1 -- 172.21.15.186:0/2613570201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc415aad0 con 0x7f1cc406da50 2024-03-28T15:01:42.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.509+0000 7f1cc9d94640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1cc406fb90 0x7f1cc4156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:42.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.510+0000 7f1cc8d92640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc406da50 0x7f1cc414fda0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800c440 tx=0x7f1cb800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:42.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.510+0000 7f1cc1ffb640 1 -- 172.21.15.186:0/2613570201 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8017070 con 0x7f1cc406da50 2024-03-28T15:01:42.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.510+0000 7f1cc1ffb640 1 -- 172.21.15.186:0/2613570201 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cb8005ce0 con 0x7f1cc406da50 2024-03-28T15:01:42.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.510+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/2613570201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc40c54e0 con 0x7f1cc406da50 2024-03-28T15:01:42.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.510+0000 7f1cc1ffb640 1 -- 172.21.15.186:0/2613570201 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8007270 con 0x7f1cc406da50 2024-03-28T15:01:42.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.510+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/2613570201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc40c6990 con 0x7f1cc406da50 2024-03-28T15:01:42.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.511+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/2613570201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc4063180 con 0x7f1cc406da50 2024-03-28T15:01:42.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.511+0000 7f1cc1ffb640 1 -- 172.21.15.186:0/2613570201 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1cb801a070 con 0x7f1cc406da50 2024-03-28T15:01:42.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.512+0000 7f1cc1ffb640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1cb006c530 0x7f1cb006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.512+0000 7f1cc1ffb640 1 -- 172.21.15.186:0/2613570201 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1cb808a860 con 0x7f1cc406da50 2024-03-28T15:01:42.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.512+0000 7f1cc9593640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1cb006c530 0x7f1cb006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.512+0000 7f1cc9593640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1cb006c530 0x7f1cb006e9f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4036000 tx=0x7f1cb4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:42.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.516+0000 7f1cc1ffb640 1 -- 172.21.15.186:0/2613570201 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f1cc4063180 con 0x7f1cc406da50 2024-03-28T15:01:42.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.728+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/2613570201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cc40686d0 con 0x7f1cc406da50 2024-03-28T15:01:42.730 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.729+0000 7f1cc1ffb640 1 -- 172.21.15.186:0/2613570201 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7f1cb808a020 con 0x7f1cc406da50 2024-03-28T15:01:42.730 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:42.730 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:01:42.730 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:01:42.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/2613570201 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1cb006c530 msgr2=0x7f1cb006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1cb006c530 0x7f1cb006e9f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4036000 tx=0x7f1cb4030000 comp rx=0 tx=0).stop 2024-03-28T15:01:42.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/2613570201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc406da50 msgr2=0x7f1cc414fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc406da50 0x7f1cc414fda0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800c440 tx=0x7f1cb800c900 comp rx=0 tx=0).stop 2024-03-28T15:01:42.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/2613570201 shutdown_connections 2024-03-28T15:01:42.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1cb006c530 0x7f1cb006e9f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1cc406fb90 0x7f1cc4156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc406da50 0x7f1cc414fda0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 --2- 172.21.15.186:0/2613570201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1cc406d670 0x7f1cc414ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/2613570201 >> 172.21.15.186:0/2613570201 conn(0x7f1cc4067670 msgr2=0x7f1cc40cc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:42.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/2613570201 shutdown_connections 2024-03-28T15:01:42.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.731+0000 7f1ccb7f7640 1 -- 172.21.15.186:0/2613570201 wait complete. 2024-03-28T15:01:42.743 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:42.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.843+0000 7fa5fcf25640 1 Processor -- start 2024-03-28T15:01:42.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.843+0000 7fa5fcf25640 1 -- start start 2024-03-28T15:01:42.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5fcf25640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5f80b1e30 0x7fa5f80712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5fcf25640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5f80c2d00 0x7fa5f80717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5fcf25640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5f806caa0 con 0x7fa5f8151990 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5fcf25640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa5f80c1da0 con 0x7fa5f80b1e30 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5fcf25640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa5f806cd80 con 0x7fa5f80c2d00 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5f5d9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5f80b1e30 0x7fa5f80712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5f5d9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5f80b1e30 0x7fa5f80712b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48548/0 (socket says 172.21.15.186:48548) 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5f5d9b640 1 -- 172.21.15.186:0/3154732395 learned_addr learned my addr 172.21.15.186:0/3154732395 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5f6d9d640 1 --2- 172.21.15.186:0/3154732395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5f80c2d00 0x7fa5f80717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5f5d9b640 1 -- 172.21.15.186:0/3154732395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5f80c2d00 msgr2=0x7fa5f80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5f5d9b640 1 --2- 172.21.15.186:0/3154732395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5f80c2d00 0x7fa5f80717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5f5d9b640 1 -- 172.21.15.186:0/3154732395 >> v1:172.21.15.191:6789/0 conn(0x7fa5f8151990 legacy=0x7fa5f80c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.844+0000 7fa5f5d9b640 1 -- 172.21.15.186:0/3154732395 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f80c1340 con 0x7fa5f80b1e30 2024-03-28T15:01:42.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.845+0000 7fa5f5d9b640 1 --2- 172.21.15.186:0/3154732395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f80b1e30 0x7fa5f80712b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009870 tx=0x7fa5e002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c39bd44d493257 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:42.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.845+0000 7fa5f559a640 1 -- 172.21.15.186:0/3154732395 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e002f840 con 0x7fa5f80b1e30 2024-03-28T15:01:42.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.845+0000 7fa5f559a640 1 -- 172.21.15.186:0/3154732395 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5e002f9e0 con 0x7fa5f80b1e30 2024-03-28T15:01:42.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.845+0000 7fa5f559a640 1 -- 172.21.15.186:0/3154732395 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e002fdb0 con 0x7fa5f80b1e30 2024-03-28T15:01:42.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.846+0000 7fa5fcf25640 1 -- 172.21.15.186:0/3154732395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f80b1e30 msgr2=0x7fa5f80712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.846+0000 7fa5fcf25640 1 --2- 172.21.15.186:0/3154732395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f80b1e30 0x7fa5f80712b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009870 tx=0x7fa5e002ba20 comp rx=0 tx=0).stop 2024-03-28T15:01:42.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.846+0000 7fa5fcf25640 1 -- 172.21.15.186:0/3154732395 shutdown_connections 2024-03-28T15:01:42.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.846+0000 7fa5fcf25640 1 --2- 172.21.15.186:0/3154732395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5f80c2d00 0x7fa5f80717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.846+0000 7fa5fcf25640 1 --2- 172.21.15.186:0/3154732395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f80b1e30 0x7fa5f80712b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.846+0000 7fa5fcf25640 1 -- 172.21.15.186:0/3154732395 >> 172.21.15.186:0/3154732395 conn(0x7fa5f80c8a20 msgr2=0x7fa5f80c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:42.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.846+0000 7fa5fcf25640 1 -- 172.21.15.186:0/3154732395 shutdown_connections 2024-03-28T15:01:42.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.846+0000 7fa5fcf25640 1 -- 172.21.15.186:0/3154732395 wait complete. 2024-03-28T15:01:42.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.847+0000 7fa5fcf25640 1 Processor -- start 2024-03-28T15:01:42.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.847+0000 7fa5fcf25640 1 -- start start 2024-03-28T15:01:42.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5fcf25640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5f80b1e30 0x7fa5f814c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5fcf25640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5f80c2d00 0x7fa5f80c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5fcf25640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f8151990 0x7fa5f80c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5fcf25640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f8071d30 con 0x7fa5f8151990 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5fcf25640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f80c1900 con 0x7fa5f80b1e30 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5fcf25640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa5f80c2940 con 0x7fa5f80c2d00 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f6d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f8151990 0x7fa5f80c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f6d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f8151990 0x7fa5f80c4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48550/0 (socket says 172.21.15.186:48550) 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f6d9d640 1 -- 172.21.15.186:0/969428425 learned_addr learned my addr 172.21.15.186:0/969428425 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f5d9b640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5f80c2d00 0x7fa5f80c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f659c640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5f80b1e30 0x7fa5f814c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f6d9d640 1 -- 172.21.15.186:0/969428425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5f80c2d00 msgr2=0x7fa5f80c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f6d9d640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5f80c2d00 0x7fa5f80c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f6d9d640 1 -- 172.21.15.186:0/969428425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5f80b1e30 msgr2=0x7fa5f814c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f6d9d640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5f80b1e30 0x7fa5f814c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:42.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f6d9d640 1 -- 172.21.15.186:0/969428425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f81f2eb0 con 0x7fa5f8151990 2024-03-28T15:01:42.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f5d9b640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5f80c2d00 0x7fa5f80c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:01:42.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5f6d9d640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f8151990 0x7fa5f80c4940 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000e7f0 tx=0x7fa5f000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:42.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5eeffd640 1 -- 172.21.15.186:0/969428425 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0017070 con 0x7fa5f8151990 2024-03-28T15:01:42.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5eeffd640 1 -- 172.21.15.186:0/969428425 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f000ee40 con 0x7fa5f8151990 2024-03-28T15:01:42.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.848+0000 7fa5fcf25640 1 -- 172.21.15.186:0/969428425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f81eeda0 con 0x7fa5f8151990 2024-03-28T15:01:42.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.849+0000 7fa5eeffd640 1 -- 172.21.15.186:0/969428425 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f000cb80 con 0x7fa5f8151990 2024-03-28T15:01:42.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.849+0000 7fa5fcf25640 1 -- 172.21.15.186:0/969428425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f81ef360 con 0x7fa5f8151990 2024-03-28T15:01:42.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.850+0000 7fa5fcf25640 1 -- 172.21.15.186:0/969428425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f81508f0 con 0x7fa5f8151990 2024-03-28T15:01:42.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.850+0000 7fa5eeffd640 1 -- 172.21.15.186:0/969428425 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa5f001a070 con 0x7fa5f8151990 2024-03-28T15:01:42.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.850+0000 7fa5eeffd640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa5d806c530 0x7fa5d806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:42.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.851+0000 7fa5f659c640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa5d806c530 0x7fa5d806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:42.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.851+0000 7fa5eeffd640 1 -- 172.21.15.186:0/969428425 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa5f008b900 con 0x7fa5f8151990 2024-03-28T15:01:42.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.851+0000 7fa5f659c640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa5d806c530 0x7fa5d806e9f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8001e70 tx=0x7fa5e8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:42.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:42.855+0000 7fa5eeffd640 1 -- 172.21.15.186:0/969428425 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa5f81508f0 con 0x7fa5f8151990 2024-03-28T15:01:43.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.030+0000 7fa5fcf25640 1 -- 172.21.15.186:0/969428425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5f8069500 con 0x7fa5f8151990 2024-03-28T15:01:43.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.031+0000 7fa5eeffd640 1 -- 172.21.15.186:0/969428425 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7fa5f0059840 con 0x7fa5f8151990 2024-03-28T15:01:43.033 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:43.033 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:33.099254+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.033+0000 7fa5fcf25640 1 -- 172.21.15.186:0/969428425 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa5d806c530 msgr2=0x7fa5d806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.033+0000 7fa5fcf25640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa5d806c530 0x7fa5d806e9f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8001e70 tx=0x7fa5e8009040 comp rx=0 tx=0).stop 2024-03-28T15:01:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.033+0000 7fa5fcf25640 1 -- 172.21.15.186:0/969428425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f8151990 msgr2=0x7fa5f80c4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.033+0000 7fa5fcf25640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f8151990 0x7fa5f80c4940 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000e7f0 tx=0x7fa5f000ebf0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.033+0000 7fa5fcf25640 1 -- 172.21.15.186:0/969428425 shutdown_connections 2024-03-28T15:01:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.033+0000 7fa5fcf25640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa5d806c530 0x7fa5d806e9f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.033+0000 7fa5fcf25640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f8151990 0x7fa5f80c4940 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.033+0000 7fa5fcf25640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5f80c2d00 0x7fa5f80c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.033+0000 7fa5fcf25640 1 --2- 172.21.15.186:0/969428425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5f80b1e30 0x7fa5f814c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.033+0000 7fa5fcf25640 1 -- 172.21.15.186:0/969428425 >> 172.21.15.186:0/969428425 conn(0x7fa5f80c8a20 msgr2=0x7fa5f814fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:43.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.034+0000 7fa5fcf25640 1 -- 172.21.15.186:0/969428425 shutdown_connections 2024-03-28T15:01:43.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.034+0000 7fa5fcf25640 1 -- 172.21.15.186:0/969428425 wait complete. 2024-03-28T15:01:43.046 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:01:18.661274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:01:29.512445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:01:30.388340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:01:43.046 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:01:43.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.149+0000 7fb1dd55e640 1 Processor -- start 2024-03-28T15:01:43.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.149+0000 7fb1dd55e640 1 -- start start 2024-03-28T15:01:43.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.150+0000 7fb1dd55e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1d8154a10 0x7fb1d80cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.150+0000 7fb1dd55e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d815de90 con 0x7fb1d815b2c0 2024-03-28T15:01:43.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.150+0000 7fb1dd55e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d80cb6d0 con 0x7fb1d80cce90 2024-03-28T15:01:43.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.150+0000 7fb1dd55e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb1d80c5ae0 con 0x7fb1d8154a10 2024-03-28T15:01:43.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.150+0000 7fb1d67fc640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fb1d815b2c0 0x7fb1d80c53d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51432/0 (socket says 172.21.15.186:51432) 2024-03-28T15:01:43.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.150+0000 7fb1d67fc640 1 -- 172.21.15.186:0/2855325834 learned_addr learned my addr 172.21.15.186:0/2855325834 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:43.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.150+0000 7fb1d6ffd640 1 --2- 172.21.15.186:0/2855325834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1d8154a10 0x7fb1d80cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.150+0000 7fb1d6ffd640 1 -- 172.21.15.186:0/2855325834 >> v1:172.21.15.191:6789/0 conn(0x7fb1d80cce90 legacy=0x7fb1d8165180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.150+0000 7fb1d6ffd640 1 -- 172.21.15.186:0/2855325834 >> v1:172.21.15.186:6789/0 conn(0x7fb1d815b2c0 legacy=0x7fb1d80c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.150+0000 7fb1d6ffd640 1 -- 172.21.15.186:0/2855325834 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d815e100 con 0x7fb1d8154a10 2024-03-28T15:01:43.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.151+0000 7fb1d6ffd640 1 --2- 172.21.15.186:0/2855325834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1d8154a10 0x7fb1d80cb0b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80097d0 tx=0x7fb1c802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17e4c9227b0f25a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:43.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.153+0000 7fb1d5ffb640 1 -- 172.21.15.186:0/2855325834 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8034070 con 0x7fb1d8154a10 2024-03-28T15:01:43.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.153+0000 7fb1d5ffb640 1 -- 172.21.15.186:0/2855325834 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1c8030880 con 0x7fb1d8154a10 2024-03-28T15:01:43.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.153+0000 7fb1d5ffb640 1 -- 172.21.15.186:0/2855325834 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8030c50 con 0x7fb1d8154a10 2024-03-28T15:01:43.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.153+0000 7fb1dd55e640 1 -- 172.21.15.186:0/2855325834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1d8154a10 msgr2=0x7fb1d80cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.153+0000 7fb1dd55e640 1 --2- 172.21.15.186:0/2855325834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1d8154a10 0x7fb1d80cb0b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80097d0 tx=0x7fb1c802b9c0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.153+0000 7fb1dd55e640 1 -- 172.21.15.186:0/2855325834 shutdown_connections 2024-03-28T15:01:43.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.153+0000 7fb1dd55e640 1 --2- 172.21.15.186:0/2855325834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1d8154a10 0x7fb1d80cb0b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.153+0000 7fb1dd55e640 1 -- 172.21.15.186:0/2855325834 >> 172.21.15.186:0/2855325834 conn(0x7fb1d806b640 msgr2=0x7fb1d806ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:43.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.153+0000 7fb1dd55e640 1 -- 172.21.15.186:0/2855325834 shutdown_connections 2024-03-28T15:01:43.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.153+0000 7fb1dd55e640 1 -- 172.21.15.186:0/2855325834 wait complete. 2024-03-28T15:01:43.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.155+0000 7fb1dd55e640 1 Processor -- start 2024-03-28T15:01:43.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.155+0000 7fb1dd55e640 1 -- start start 2024-03-28T15:01:43.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.155+0000 7fb1dd55e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1d80cce90 0x7fb1d81f1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.155+0000 7fb1dd55e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1d8154a10 0x7fb1d80b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.155+0000 7fb1dd55e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d815b2c0 0x7fb1d8150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.155+0000 7fb1dd55e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d80cb9b0 con 0x7fb1d815b2c0 2024-03-28T15:01:43.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.155+0000 7fb1dd55e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d815e750 con 0x7fb1d8154a10 2024-03-28T15:01:43.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.155+0000 7fb1dd55e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb1d815b6e0 con 0x7fb1d80cce90 2024-03-28T15:01:43.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.155+0000 7fb1d77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d815b2c0 0x7fb1d8150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1d77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d815b2c0 0x7fb1d8150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48566/0 (socket says 172.21.15.186:48566) 2024-03-28T15:01:43.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1d77fe640 1 -- 172.21.15.186:0/3182860055 learned_addr learned my addr 172.21.15.186:0/3182860055 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:43.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1d6ffd640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1d80cce90 0x7fb1d81f1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1d67fc640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1d8154a10 0x7fb1d80b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1d77fe640 1 -- 172.21.15.186:0/3182860055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1d80cce90 msgr2=0x7fb1d81f1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1d77fe640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1d80cce90 0x7fb1d81f1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1d77fe640 1 -- 172.21.15.186:0/3182860055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1d8154a10 msgr2=0x7fb1d80b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1d77fe640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1d8154a10 0x7fb1d80b55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1d77fe640 1 -- 172.21.15.186:0/3182860055 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d814f520 con 0x7fb1d815b2c0 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1d77fe640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d815b2c0 0x7fb1d8150c40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000c7a0 tx=0x7fb1d000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1bf7fe640 1 -- 172.21.15.186:0/3182860055 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0003b70 con 0x7fb1d815b2c0 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1bf7fe640 1 -- 172.21.15.186:0/3182860055 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1d0003d30 con 0x7fb1d815b2c0 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1dd55e640 1 -- 172.21.15.186:0/3182860055 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d814f800 con 0x7fb1d815b2c0 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1bf7fe640 1 -- 172.21.15.186:0/3182860055 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d000aa90 con 0x7fb1d815b2c0 2024-03-28T15:01:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.156+0000 7fb1dd55e640 1 -- 172.21.15.186:0/3182860055 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d8150440 con 0x7fb1d815b2c0 2024-03-28T15:01:43.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.157+0000 7fb1dd55e640 1 -- 172.21.15.186:0/3182860055 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1d8069570 con 0x7fb1d815b2c0 2024-03-28T15:01:43.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.158+0000 7fb1bf7fe640 1 -- 172.21.15.186:0/3182860055 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb1d000ac30 con 0x7fb1d815b2c0 2024-03-28T15:01:43.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.158+0000 7fb1bf7fe640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb1c406c560 0x7fb1c406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.158+0000 7fb1bf7fe640 1 -- 172.21.15.186:0/3182860055 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb1d008b490 con 0x7fb1d815b2c0 2024-03-28T15:01:43.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.158+0000 7fb1d6ffd640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb1c406c560 0x7fb1c406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.159+0000 7fb1d6ffd640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb1c406c560 0x7fb1c406ea20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8036000 tx=0x7fb1c803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:43.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.162+0000 7fb1bf7fe640 1 -- 172.21.15.186:0/3182860055 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb1d0059460 con 0x7fb1d815b2c0 2024-03-28T15:01:43.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.336+0000 7fb1dd55e640 1 -- 172.21.15.186:0/3182860055 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb1d815c960 con 0x7fb1d815b2c0 2024-03-28T15:01:43.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.336+0000 7fb1bf7fe640 1 -- 172.21.15.186:0/3182860055 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fb1d005d760 con 0x7fb1d815b2c0 2024-03-28T15:01:43.337 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 1 2024-03-28T15:01:43.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 -- 172.21.15.186:0/3182860055 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb1c406c560 msgr2=0x7fb1c406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb1c406c560 0x7fb1c406ea20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8036000 tx=0x7fb1c803b040 comp rx=0 tx=0).stop 2024-03-28T15:01:43.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 -- 172.21.15.186:0/3182860055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d815b2c0 msgr2=0x7fb1d8150c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d815b2c0 0x7fb1d8150c40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000c7a0 tx=0x7fb1d000cc60 comp rx=0 tx=0).stop 2024-03-28T15:01:43.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 -- 172.21.15.186:0/3182860055 shutdown_connections 2024-03-28T15:01:43.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb1c406c560 0x7fb1c406ea20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d815b2c0 0x7fb1d8150c40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1d8154a10 0x7fb1d80b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 --2- 172.21.15.186:0/3182860055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1d80cce90 0x7fb1d81f1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 -- 172.21.15.186:0/3182860055 >> 172.21.15.186:0/3182860055 conn(0x7fb1d806b640 msgr2=0x7fb1d80c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:43.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 -- 172.21.15.186:0/3182860055 shutdown_connections 2024-03-28T15:01:43.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.338+0000 7fb1dd55e640 1 -- 172.21.15.186:0/3182860055 wait complete. 2024-03-28T15:01:43.350 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:01:43.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.453+0000 7f09e43c7640 1 Processor -- start 2024-03-28T15:01:43.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.453+0000 7f09e43c7640 1 -- start start 2024-03-28T15:01:43.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e43c7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c7a80 0x7f09dc158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e43c7640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09dc159420 0x7f09dc06c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e43c7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09dc0725b0 con 0x7f09dc152960 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e43c7640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f09dc15bda0 con 0x7f09dc159420 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e43c7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f09dc158a50 con 0x7f09dc0c7a80 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e2964640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09dc159420 0x7f09dc06c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e1962640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c7a80 0x7f09dc158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e2163640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f09dc152960 0x7f09dc06b6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:58646/0 (socket says 172.21.15.186:58646) 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e2163640 1 -- 172.21.15.186:0/1335753071 learned_addr learned my addr 172.21.15.186:0/1335753071 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e1161640 1 -- 172.21.15.186:0/1335753071 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3004525821 0 0) 0x7f09dc0725b0 con 0x7f09dc152960 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e1161640 1 -- 172.21.15.186:0/1335753071 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09c8003520 con 0x7f09dc152960 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e2964640 1 -- 172.21.15.186:0/1335753071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c7a80 msgr2=0x7f09dc158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e2964640 1 --2- 172.21.15.186:0/1335753071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c7a80 0x7f09dc158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e2964640 1 -- 172.21.15.186:0/1335753071 >> v1:172.21.15.191:6789/0 conn(0x7f09dc152960 legacy=0x7f09dc06b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e2964640 1 -- 172.21.15.186:0/1335753071 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc072890 con 0x7f09dc159420 2024-03-28T15:01:43.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.454+0000 7f09e1962640 1 --2- 172.21.15.186:0/1335753071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c7a80 0x7f09dc158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:43.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.455+0000 7f09e2964640 1 --2- 172.21.15.186:0/1335753071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc159420 0x7f09dc06c160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009d90 tx=0x7f09d402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38cd419973ea12a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:43.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.455+0000 7f09e1161640 1 -- 172.21.15.186:0/1335753071 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4039070 con 0x7f09dc159420 2024-03-28T15:01:43.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.455+0000 7f09e1161640 1 -- 172.21.15.186:0/1335753071 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09d402cce0 con 0x7f09dc159420 2024-03-28T15:01:43.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.455+0000 7f09e1161640 1 -- 172.21.15.186:0/1335753071 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4034640 con 0x7f09dc159420 2024-03-28T15:01:43.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.455+0000 7f09e43c7640 1 -- 172.21.15.186:0/1335753071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc159420 msgr2=0x7f09dc06c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.455+0000 7f09e43c7640 1 --2- 172.21.15.186:0/1335753071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc159420 0x7f09dc06c160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009d90 tx=0x7f09d402bf30 comp rx=0 tx=0).stop 2024-03-28T15:01:43.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.456+0000 7f09e43c7640 1 -- 172.21.15.186:0/1335753071 shutdown_connections 2024-03-28T15:01:43.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.456+0000 7f09e43c7640 1 --2- 172.21.15.186:0/1335753071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc159420 0x7f09dc06c160 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.456+0000 7f09e43c7640 1 --2- 172.21.15.186:0/1335753071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c7a80 0x7f09dc158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.456+0000 7f09e43c7640 1 -- 172.21.15.186:0/1335753071 >> 172.21.15.186:0/1335753071 conn(0x7f09dc06e990 msgr2=0x7f09dc06ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:43.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.456+0000 7f09e43c7640 1 -- 172.21.15.186:0/1335753071 shutdown_connections 2024-03-28T15:01:43.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.456+0000 7f09e43c7640 1 -- 172.21.15.186:0/1335753071 wait complete. 2024-03-28T15:01:43.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.457+0000 7f09e43c7640 1 Processor -- start 2024-03-28T15:01:43.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.457+0000 7f09e43c7640 1 -- start start 2024-03-28T15:01:43.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.457+0000 7f09e43c7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c7a80 0x7f09dc14cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.457+0000 7f09e43c7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc152960 0x7f09dc0c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.457+0000 7f09e43c7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc159420 0x7f09dc0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.457+0000 7f09e43c7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc073cf0 con 0x7f09dc159420 2024-03-28T15:01:43.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.457+0000 7f09e43c7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc072e50 con 0x7f09dc0c7a80 2024-03-28T15:01:43.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.457+0000 7f09e43c7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f09dc073e70 con 0x7f09dc152960 2024-03-28T15:01:43.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e2964640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc159420 0x7f09dc0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e2163640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c7a80 0x7f09dc14cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e2163640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c7a80 0x7f09dc14cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:37398/0 (socket says 172.21.15.186:37398) 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e1962640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc152960 0x7f09dc0c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e2964640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc159420 0x7f09dc0ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48580/0 (socket says 172.21.15.186:48580) 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e2163640 1 -- 172.21.15.186:0/2211708178 learned_addr learned my addr 172.21.15.186:0/2211708178 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e1962640 1 -- 172.21.15.186:0/2211708178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c7a80 msgr2=0x7f09dc14cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e1962640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c7a80 0x7f09dc14cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e1962640 1 -- 172.21.15.186:0/2211708178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc159420 msgr2=0x7f09dc0ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e1962640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc159420 0x7f09dc0ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e1962640 1 -- 172.21.15.186:0/2211708178 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc0c3e40 con 0x7f09dc152960 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e2163640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c7a80 0x7f09dc14cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e1962640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc152960 0x7f09dc0c4440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00e900 tx=0x7f09cc00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09d2ffd640 1 -- 172.21.15.186:0/2211708178 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc017070 con 0x7f09dc152960 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09d2ffd640 1 -- 172.21.15.186:0/2211708178 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09cc00ca00 con 0x7f09dc152960 2024-03-28T15:01:43.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09d2ffd640 1 -- 172.21.15.186:0/2211708178 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc00cdf0 con 0x7f09dc152960 2024-03-28T15:01:43.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e43c7640 1 -- 172.21.15.186:0/2211708178 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09dc0c6eb0 con 0x7f09dc152960 2024-03-28T15:01:43.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.458+0000 7f09e43c7640 1 -- 172.21.15.186:0/2211708178 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f09dc0c7470 con 0x7f09dc152960 2024-03-28T15:01:43.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.459+0000 7f09e43c7640 1 -- 172.21.15.186:0/2211708178 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09dc0c8c30 con 0x7f09dc152960 2024-03-28T15:01:43.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.460+0000 7f09d2ffd640 1 -- 172.21.15.186:0/2211708178 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f09cc01a070 con 0x7f09dc152960 2024-03-28T15:01:43.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.460+0000 7f09d2ffd640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c006c530 0x7f09c006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.460+0000 7f09e2163640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c006c530 0x7f09c006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.460+0000 7f09d2ffd640 1 -- 172.21.15.186:0/2211708178 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f09cc08b860 con 0x7f09dc152960 2024-03-28T15:01:43.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.460+0000 7f09e2163640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c006c530 0x7f09c006e9f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f09d8005880 tx=0x7f09d8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:43.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.465+0000 7f09d2ffd640 1 -- 172.21.15.186:0/2211708178 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f09dc0c8c30 con 0x7f09dc152960 2024-03-28T15:01:43.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.639+0000 7f09e43c7640 1 -- 172.21.15.186:0/2211708178 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f09dc15c3a0 con 0x7f09dc152960 2024-03-28T15:01:43.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.639+0000 7f09d2ffd640 1 -- 172.21.15.186:0/2211708178 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f09dc15c3a0 con 0x7f09dc152960 2024-03-28T15:01:43.641 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 32 2024-03-28T15:01:43.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.641+0000 7f09e43c7640 1 -- 172.21.15.186:0/2211708178 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c006c530 msgr2=0x7f09c006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.641+0000 7f09e43c7640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c006c530 0x7f09c006e9f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f09d8005880 tx=0x7f09d8009040 comp rx=0 tx=0).stop 2024-03-28T15:01:43.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.641+0000 7f09e43c7640 1 -- 172.21.15.186:0/2211708178 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc152960 msgr2=0x7f09dc0c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.641+0000 7f09e43c7640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc152960 0x7f09dc0c4440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00e900 tx=0x7f09cc00edc0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.641+0000 7f09e43c7640 1 -- 172.21.15.186:0/2211708178 shutdown_connections 2024-03-28T15:01:43.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.641+0000 7f09e43c7640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c006c530 0x7f09c006e9f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.641+0000 7f09e43c7640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc159420 0x7f09dc0ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.641+0000 7f09e43c7640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc152960 0x7f09dc0c4440 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.641+0000 7f09e43c7640 1 --2- 172.21.15.186:0/2211708178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c7a80 0x7f09dc14cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.641+0000 7f09e43c7640 1 -- 172.21.15.186:0/2211708178 >> 172.21.15.186:0/2211708178 conn(0x7f09dc06e990 msgr2=0x7f09dc06be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:43.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.642+0000 7f09e43c7640 1 -- 172.21.15.186:0/2211708178 shutdown_connections 2024-03-28T15:01:43.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.642+0000 7f09e43c7640 1 -- 172.21.15.186:0/2211708178 wait complete. 2024-03-28T15:01:43.654 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:01:43.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.757+0000 7f22473c9640 1 Processor -- start 2024-03-28T15:01:43.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.757+0000 7f22473c9640 1 -- start start 2024-03-28T15:01:43.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.758+0000 7f22473c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2240154a10 0x7f22400cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.758+0000 7f22473c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22400cce90 0x7f22400cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.758+0000 7f22473c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f224015de90 con 0x7f224015b2c0 2024-03-28T15:01:43.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.758+0000 7f22473c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f22400c5ae0 con 0x7f2240154a10 2024-03-28T15:01:43.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.758+0000 7f22473c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f224015e750 con 0x7f22400cce90 2024-03-28T15:01:43.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.758+0000 7f2245165640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2240154a10 0x7f22400cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.758+0000 7f2245165640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2240154a10 0x7f22400cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48594/0 (socket says 172.21.15.186:48594) 2024-03-28T15:01:43.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.758+0000 7f2245165640 1 -- 172.21.15.186:0/1821129114 learned_addr learned my addr 172.21.15.186:0/1821129114 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:43.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.758+0000 7f2245966640 1 --2- 172.21.15.186:0/1821129114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22400cce90 0x7f22400cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.759+0000 7f2245165640 1 -- 172.21.15.186:0/1821129114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22400cce90 msgr2=0x7f22400cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:43.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.759+0000 7f2245165640 1 --2- 172.21.15.186:0/1821129114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22400cce90 0x7f22400cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.759+0000 7f2245165640 1 -- 172.21.15.186:0/1821129114 >> v1:172.21.15.186:6789/0 conn(0x7f224015b2c0 legacy=0x7f22400c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.759+0000 7f2245165640 1 -- 172.21.15.186:0/1821129114 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22400cb6d0 con 0x7f2240154a10 2024-03-28T15:01:43.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.759+0000 7f2245966640 1 --2- 172.21.15.186:0/1821129114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22400cce90 0x7f22400cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:43.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.759+0000 7f2245165640 1 --2- 172.21.15.186:0/1821129114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2240154a10 0x7f22400cb0b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f223c009870 tx=0x7f223c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a71254d6acdca74 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:43.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.760+0000 7f2237fff640 1 -- 172.21.15.186:0/1821129114 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c02f840 con 0x7f2240154a10 2024-03-28T15:01:43.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.760+0000 7f2237fff640 1 -- 172.21.15.186:0/1821129114 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f223c02f9e0 con 0x7f2240154a10 2024-03-28T15:01:43.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.760+0000 7f2237fff640 1 -- 172.21.15.186:0/1821129114 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c02fdb0 con 0x7f2240154a10 2024-03-28T15:01:43.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.760+0000 7f22473c9640 1 -- 172.21.15.186:0/1821129114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2240154a10 msgr2=0x7f22400cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.760+0000 7f22473c9640 1 --2- 172.21.15.186:0/1821129114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2240154a10 0x7f22400cb0b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f223c009870 tx=0x7f223c02ba20 comp rx=0 tx=0).stop 2024-03-28T15:01:43.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.760+0000 7f22473c9640 1 -- 172.21.15.186:0/1821129114 shutdown_connections 2024-03-28T15:01:43.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.760+0000 7f22473c9640 1 --2- 172.21.15.186:0/1821129114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22400cce90 0x7f22400cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.760+0000 7f22473c9640 1 --2- 172.21.15.186:0/1821129114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2240154a10 0x7f22400cb0b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.760+0000 7f22473c9640 1 -- 172.21.15.186:0/1821129114 >> 172.21.15.186:0/1821129114 conn(0x7f224006b640 msgr2=0x7f224006ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:43.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.760+0000 7f22473c9640 1 -- 172.21.15.186:0/1821129114 shutdown_connections 2024-03-28T15:01:43.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.760+0000 7f22473c9640 1 -- 172.21.15.186:0/1821129114 wait complete. 2024-03-28T15:01:43.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.761+0000 7f22473c9640 1 Processor -- start 2024-03-28T15:01:43.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.761+0000 7f22473c9640 1 -- start start 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f22473c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22400cce90 0x7f224014c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f22473c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2240154a10 0x7f22401f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f22473c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224015b2c0 0x7f22401f0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f22473c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22400cb9b0 con 0x7f224015b2c0 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f22473c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f224015e3e0 con 0x7f22400cce90 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f22473c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f224015b710 con 0x7f2240154a10 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f2245966640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224015b2c0 0x7f22401f0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f2245966640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224015b2c0 0x7f22401f0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48600/0 (socket says 172.21.15.186:48600) 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f2245966640 1 -- 172.21.15.186:0/1355205752 learned_addr learned my addr 172.21.15.186:0/1355205752 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f2244964640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2240154a10 0x7f22401f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f2245165640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22400cce90 0x7f224014c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f2245966640 1 -- 172.21.15.186:0/1355205752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2240154a10 msgr2=0x7f22401f06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f2245966640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2240154a10 0x7f22401f06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f2245966640 1 -- 172.21.15.186:0/1355205752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22400cce90 msgr2=0x7f224014c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f2245966640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22400cce90 0x7f224014c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.762+0000 7f2245966640 1 -- 172.21.15.186:0/1355205752 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f224014d020 con 0x7f224015b2c0 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.763+0000 7f2245966640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224015b2c0 0x7f22401f0c10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2238009f00 tx=0x7f223800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.763+0000 7f2235ffb640 1 -- 172.21.15.186:0/1355205752 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2238018030 con 0x7f224015b2c0 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.763+0000 7f2235ffb640 1 -- 172.21.15.186:0/1355205752 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f223800ce50 con 0x7f224015b2c0 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.763+0000 7f22473c9640 1 -- 172.21.15.186:0/1355205752 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22401f3de0 con 0x7f224015b2c0 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.763+0000 7f2235ffb640 1 -- 172.21.15.186:0/1355205752 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2238003bb0 con 0x7f224015b2c0 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.763+0000 7f22473c9640 1 -- 172.21.15.186:0/1355205752 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22401f43a0 con 0x7f224015b2c0 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.764+0000 7f22473c9640 1 -- 172.21.15.186:0/1355205752 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2240069570 con 0x7f224015b2c0 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.764+0000 7f2235ffb640 1 -- 172.21.15.186:0/1355205752 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f223801b070 con 0x7f224015b2c0 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.765+0000 7f2235ffb640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f222406c530 0x7f222406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.765+0000 7f2245165640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f222406c530 0x7f222406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.765+0000 7f2235ffb640 1 -- 172.21.15.186:0/1355205752 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f223808bf70 con 0x7f224015b2c0 2024-03-28T15:01:43.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.765+0000 7f2245165640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f222406c530 0x7f222406e9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f223c036000 tx=0x7f223c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:43.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.769+0000 7f2235ffb640 1 -- 172.21.15.186:0/1355205752 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f2238010040 con 0x7f224015b2c0 2024-03-28T15:01:43.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.944+0000 7f22473c9640 1 -- 172.21.15.186:0/1355205752 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f224014ea90 con 0x7f224015b2c0 2024-03-28T15:01:43.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.944+0000 7f2235ffb640 1 -- 172.21.15.186:0/1355205752 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f2238059fd0 con 0x7f224015b2c0 2024-03-28T15:01:43.946 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 64 2024-03-28T15:01:43.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 -- 172.21.15.186:0/1355205752 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f222406c530 msgr2=0x7f222406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f222406c530 0x7f222406e9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f223c036000 tx=0x7f223c008030 comp rx=0 tx=0).stop 2024-03-28T15:01:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 -- 172.21.15.186:0/1355205752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224015b2c0 msgr2=0x7f22401f0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224015b2c0 0x7f22401f0c10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2238009f00 tx=0x7f223800c490 comp rx=0 tx=0).stop 2024-03-28T15:01:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 -- 172.21.15.186:0/1355205752 shutdown_connections 2024-03-28T15:01:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f222406c530 0x7f222406e9f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224015b2c0 0x7f22401f0c10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2240154a10 0x7f22401f06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 --2- 172.21.15.186:0/1355205752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22400cce90 0x7f224014c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 -- 172.21.15.186:0/1355205752 >> 172.21.15.186:0/1355205752 conn(0x7f224006b640 msgr2=0x7f224015d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 -- 172.21.15.186:0/1355205752 shutdown_connections 2024-03-28T15:01:43.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:43.946+0000 7f22473c9640 1 -- 172.21.15.186:0/1355205752 wait complete. 2024-03-28T15:01:43.958 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:01:44.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.058+0000 7fd7f3125640 1 Processor -- start 2024-03-28T15:01:44.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.058+0000 7fd7f3125640 1 -- start start 2024-03-28T15:01:44.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.059+0000 7fd7f3125640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06d670 0x7fd7ec06f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.059+0000 7fd7f3125640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd7ec06da50 0x7fd7ec06dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.059+0000 7fd7f3125640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ec06f770 con 0x7fd7ec06fb90 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.059+0000 7fd7f3125640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ec0c91a0 con 0x7fd7ec06da50 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.059+0000 7fd7f3125640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ec0c4c60 con 0x7fd7ec06d670 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.059+0000 7fd7f16c2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd7ec06da50 0x7fd7ec06dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.059+0000 7fd7f0ec1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06d670 0x7fd7ec06f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.059+0000 7fd7f0ec1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06d670 0x7fd7ec06f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:51936/0 (socket says 172.21.15.186:51936) 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.059+0000 7fd7f16c2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd7ec06da50 0x7fd7ec06dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48604/0 (socket says 172.21.15.186:48604) 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.059+0000 7fd7f0ec1640 1 -- 172.21.15.186:0/1620348283 learned_addr learned my addr 172.21.15.186:0/1620348283 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.060+0000 7fd7f0ec1640 1 -- 172.21.15.186:0/1620348283 >> v2:172.21.15.186:3300/0 conn(0x7fd7ec06da50 msgr2=0x7fd7ec06dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.060+0000 7fd7f0ec1640 1 --2- 172.21.15.186:0/1620348283 >> v2:172.21.15.186:3300/0 conn(0x7fd7ec06da50 0x7fd7ec06dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.060+0000 7fd7f0ec1640 1 -- 172.21.15.186:0/1620348283 >> v1:172.21.15.191:6789/0 conn(0x7fd7ec06fb90 legacy=0x7fd7ec0c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.060+0000 7fd7f0ec1640 1 -- 172.21.15.186:0/1620348283 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec159500 con 0x7fd7ec06d670 2024-03-28T15:01:44.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.060+0000 7fd7eb7fe640 1 -- 172.21.15.186:0/1620348283 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1160854522 0 0) 0x7fd7ec06f770 con 0x7fd7ec06fb90 2024-03-28T15:01:44.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.060+0000 7fd7f0ec1640 1 --2- 172.21.15.186:0/1620348283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06d670 0x7fd7ec06f0a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009e50 tx=0x7fd7e402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=981399e5fd70eb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:44.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.061+0000 7fd7eb7fe640 1 -- 172.21.15.186:0/1620348283 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e402ca70 con 0x7fd7ec06d670 2024-03-28T15:01:44.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.061+0000 7fd7eb7fe640 1 -- 172.21.15.186:0/1620348283 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e402cc10 con 0x7fd7ec06d670 2024-03-28T15:01:44.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.061+0000 7fd7eb7fe640 1 -- 172.21.15.186:0/1620348283 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40343f0 con 0x7fd7ec06d670 2024-03-28T15:01:44.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.061+0000 7fd7f3125640 1 -- 172.21.15.186:0/1620348283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06d670 msgr2=0x7fd7ec06f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.061+0000 7fd7f3125640 1 --2- 172.21.15.186:0/1620348283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06d670 0x7fd7ec06f0a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009e50 tx=0x7fd7e402bed0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.061+0000 7fd7f3125640 1 -- 172.21.15.186:0/1620348283 shutdown_connections 2024-03-28T15:01:44.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.061+0000 7fd7f3125640 1 --2- 172.21.15.186:0/1620348283 >> v2:172.21.15.186:3300/0 conn(0x7fd7ec06da50 0x7fd7ec06dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.061+0000 7fd7f3125640 1 --2- 172.21.15.186:0/1620348283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06d670 0x7fd7ec06f0a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.061+0000 7fd7f3125640 1 -- 172.21.15.186:0/1620348283 >> 172.21.15.186:0/1620348283 conn(0x7fd7ec067670 msgr2=0x7fd7ec071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.061+0000 7fd7f3125640 1 -- 172.21.15.186:0/1620348283 shutdown_connections 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.061+0000 7fd7f3125640 1 -- 172.21.15.186:0/1620348283 wait complete. 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.062+0000 7fd7f3125640 1 Processor -- start 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.062+0000 7fd7f3125640 1 -- start start 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f3125640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec06d670 0x7fd7ec1e53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f3125640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06da50 0x7fd7ec153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f3125640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7ec06fb90 0x7fd7ec154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f3125640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec0c34c0 con 0x7fd7ec06d670 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f3125640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec0c31e0 con 0x7fd7ec06fb90 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f3125640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ec0c50f0 con 0x7fd7ec06da50 2024-03-28T15:01:44.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f0ec1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec06d670 0x7fd7ec1e53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f0ec1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec06d670 0x7fd7ec1e53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48610/0 (socket says 172.21.15.186:48610) 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f0ec1640 1 -- 172.21.15.186:0/1859526227 learned_addr learned my addr 172.21.15.186:0/1859526227 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7ebfff640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06da50 0x7fd7ec153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f16c2640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7ec06fb90 0x7fd7ec154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f0ec1640 1 -- 172.21.15.186:0/1859526227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06da50 msgr2=0x7fd7ec153b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.063+0000 7fd7f0ec1640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06da50 0x7fd7ec153b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.064+0000 7fd7f0ec1640 1 -- 172.21.15.186:0/1859526227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7ec06fb90 msgr2=0x7fd7ec154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.064+0000 7fd7f0ec1640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7ec06fb90 0x7fd7ec154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.064+0000 7fd7f0ec1640 1 -- 172.21.15.186:0/1859526227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec156650 con 0x7fd7ec06d670 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.064+0000 7fd7f0ec1640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec06d670 0x7fd7ec1e53d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4000f80 tx=0x7fd7e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.064+0000 7fd7e97fa640 1 -- 172.21.15.186:0/1859526227 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4043020 con 0x7fd7ec06d670 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.064+0000 7fd7e97fa640 1 -- 172.21.15.186:0/1859526227 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e403e450 con 0x7fd7ec06d670 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.064+0000 7fd7e97fa640 1 -- 172.21.15.186:0/1859526227 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4009510 con 0x7fd7ec06d670 2024-03-28T15:01:44.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.064+0000 7fd7f3125640 1 -- 172.21.15.186:0/1859526227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ec156930 con 0x7fd7ec06d670 2024-03-28T15:01:44.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.064+0000 7fd7f3125640 1 -- 172.21.15.186:0/1859526227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ec156ef0 con 0x7fd7ec06d670 2024-03-28T15:01:44.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.065+0000 7fd7f3125640 1 -- 172.21.15.186:0/1859526227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ec0686d0 con 0x7fd7ec06d670 2024-03-28T15:01:44.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.065+0000 7fd7e97fa640 1 -- 172.21.15.186:0/1859526227 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd7e4046020 con 0x7fd7ec06d670 2024-03-28T15:01:44.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.066+0000 7fd7e97fa640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7cc06c530 0x7fd7cc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.066+0000 7fd7e97fa640 1 -- 172.21.15.186:0/1859526227 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd7e40addb0 con 0x7fd7ec06d670 2024-03-28T15:01:44.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.066+0000 7fd7ebfff640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7cc06c530 0x7fd7cc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.066+0000 7fd7ebfff640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7cc06c530 0x7fd7cc06e9f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec1f23a0 tx=0x7fd7dc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:44.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.070+0000 7fd7e97fa640 1 -- 172.21.15.186:0/1859526227 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd7e404d030 con 0x7fd7ec06d670 2024-03-28T15:01:44.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.284+0000 7fd7f3125640 1 -- 172.21.15.186:0/1859526227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7ec06f970 con 0x7fd7ec06d670 2024-03-28T15:01:44.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.284+0000 7fd7e97fa640 1 -- 172.21.15.186:0/1859526227 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7fd7e407bd00 con 0x7fd7ec06d670 2024-03-28T15:01:44.286 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:44.286 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:01:44.286 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:01:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 -- 172.21.15.186:0/1859526227 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7cc06c530 msgr2=0x7fd7cc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7cc06c530 0x7fd7cc06e9f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec1f23a0 tx=0x7fd7dc009040 comp rx=0 tx=0).stop 2024-03-28T15:01:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 -- 172.21.15.186:0/1859526227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec06d670 msgr2=0x7fd7ec1e53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec06d670 0x7fd7ec1e53d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4000f80 tx=0x7fd7e4004a40 comp rx=0 tx=0).stop 2024-03-28T15:01:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 -- 172.21.15.186:0/1859526227 shutdown_connections 2024-03-28T15:01:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7cc06c530 0x7fd7cc06e9f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7ec06fb90 0x7fd7ec154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7ec06da50 0x7fd7ec153b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 --2- 172.21.15.186:0/1859526227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7ec06d670 0x7fd7ec1e53d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 -- 172.21.15.186:0/1859526227 >> 172.21.15.186:0/1859526227 conn(0x7fd7ec067670 msgr2=0x7fd7ec151e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 -- 172.21.15.186:0/1859526227 shutdown_connections 2024-03-28T15:01:44.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.286+0000 7fd7f3125640 1 -- 172.21.15.186:0/1859526227 wait complete. 2024-03-28T15:01:44.299 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:44.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.398+0000 7f10b85ac640 1 Processor -- start 2024-03-28T15:01:44.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.398+0000 7f10b85ac640 1 -- start start 2024-03-28T15:01:44.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b85ac640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10b0154bd0 0x7f10b015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b85ac640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f10b00c57e0 0x7f10b00cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b85ac640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b00c6e00 con 0x7f10b00c53c0 2024-03-28T15:01:44.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b85ac640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f10b00c5ca0 con 0x7f10b0154bd0 2024-03-28T15:01:44.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b85ac640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f10b015b080 con 0x7f10b00c57e0 2024-03-28T15:01:44.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b6348640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10b0154bd0 0x7f10b015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b6348640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10b0154bd0 0x7f10b015a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48624/0 (socket says 172.21.15.186:48624) 2024-03-28T15:01:44.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b6348640 1 -- 172.21.15.186:0/2620335553 learned_addr learned my addr 172.21.15.186:0/2620335553 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:44.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b6b49640 1 --2- 172.21.15.186:0/2620335553 >> v2:172.21.15.191:3300/0 conn(0x7f10b00c57e0 0x7f10b00cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b6348640 1 -- 172.21.15.186:0/2620335553 >> v2:172.21.15.191:3300/0 conn(0x7f10b00c57e0 msgr2=0x7f10b00cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:44.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b6348640 1 --2- 172.21.15.186:0/2620335553 >> v2:172.21.15.191:3300/0 conn(0x7f10b00c57e0 0x7f10b00cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b6348640 1 -- 172.21.15.186:0/2620335553 >> v1:172.21.15.186:6789/0 conn(0x7f10b00c53c0 legacy=0x7f10b00c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b6348640 1 -- 172.21.15.186:0/2620335553 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b015b360 con 0x7f10b0154bd0 2024-03-28T15:01:44.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.399+0000 7f10b6b49640 1 --2- 172.21.15.186:0/2620335553 >> v2:172.21.15.191:3300/0 conn(0x7f10b00c57e0 0x7f10b00cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:44.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.400+0000 7f10b6348640 1 --2- 172.21.15.186:0/2620335553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0154bd0 0x7f10b015a7d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f10b00b4180 tx=0x7f10ac02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516578fd74400bde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:44.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.400+0000 7f10b5346640 1 -- 172.21.15.186:0/2620335553 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac02f850 con 0x7f10b0154bd0 2024-03-28T15:01:44.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.400+0000 7f10b5346640 1 -- 172.21.15.186:0/2620335553 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ac02f9f0 con 0x7f10b0154bd0 2024-03-28T15:01:44.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.400+0000 7f10b5346640 1 -- 172.21.15.186:0/2620335553 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac02fdc0 con 0x7f10b0154bd0 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.401+0000 7f10b85ac640 1 -- 172.21.15.186:0/2620335553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0154bd0 msgr2=0x7f10b015a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.401+0000 7f10b85ac640 1 --2- 172.21.15.186:0/2620335553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0154bd0 0x7f10b015a7d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f10b00b4180 tx=0x7f10ac02ba30 comp rx=0 tx=0).stop 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.401+0000 7f10b85ac640 1 -- 172.21.15.186:0/2620335553 shutdown_connections 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.401+0000 7f10b85ac640 1 --2- 172.21.15.186:0/2620335553 >> v2:172.21.15.191:3300/0 conn(0x7f10b00c57e0 0x7f10b00cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.401+0000 7f10b85ac640 1 --2- 172.21.15.186:0/2620335553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b0154bd0 0x7f10b015a7d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.401+0000 7f10b85ac640 1 -- 172.21.15.186:0/2620335553 >> 172.21.15.186:0/2620335553 conn(0x7f10b00731f0 msgr2=0x7f10b0073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.401+0000 7f10b85ac640 1 -- 172.21.15.186:0/2620335553 shutdown_connections 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.401+0000 7f10b85ac640 1 -- 172.21.15.186:0/2620335553 wait complete. 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.402+0000 7f10b85ac640 1 Processor -- start 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.402+0000 7f10b85ac640 1 -- start start 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b85ac640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10b00c53c0 0x7f10b01585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b85ac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b00c57e0 0x7f10b0158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b85ac640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10b0154bd0 0x7f10b01f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b85ac640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f10b00c61d0 con 0x7f10b00c57e0 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b85ac640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f10b00c5ef0 con 0x7f10b0154bd0 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b85ac640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f10b00cc390 con 0x7f10b00c53c0 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b5b47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b00c57e0 0x7f10b0158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b5b47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b00c57e0 0x7f10b0158b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48628/0 (socket says 172.21.15.186:48628) 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b5b47640 1 -- 172.21.15.186:0/3112961669 learned_addr learned my addr 172.21.15.186:0/3112961669 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b6348640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10b00c53c0 0x7f10b01585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b6b49640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10b0154bd0 0x7f10b01f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b5b47640 1 -- 172.21.15.186:0/3112961669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10b00c53c0 msgr2=0x7f10b01585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b5b47640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10b00c53c0 0x7f10b01585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b5b47640 1 -- 172.21.15.186:0/3112961669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10b0154bd0 msgr2=0x7f10b01f3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b5b47640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10b0154bd0 0x7f10b01f3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b5b47640 1 -- 172.21.15.186:0/3112961669 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b01f4c40 con 0x7f10b00c57e0 2024-03-28T15:01:44.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b5b47640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b00c57e0 0x7f10b0158b30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10a000c430 tx=0x7f10a000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:44.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10a6ffd640 1 -- 172.21.15.186:0/3112961669 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a00052d0 con 0x7f10b00c57e0 2024-03-28T15:01:44.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10a6ffd640 1 -- 172.21.15.186:0/3112961669 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a0005490 con 0x7f10b00c57e0 2024-03-28T15:01:44.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b85ac640 1 -- 172.21.15.186:0/3112961669 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b015a200 con 0x7f10b00c57e0 2024-03-28T15:01:44.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10a6ffd640 1 -- 172.21.15.186:0/3112961669 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0007260 con 0x7f10b00c57e0 2024-03-28T15:01:44.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.403+0000 7f10b85ac640 1 -- 172.21.15.186:0/3112961669 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b00b4d30 con 0x7f10b00c57e0 2024-03-28T15:01:44.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.404+0000 7f10b85ac640 1 -- 172.21.15.186:0/3112961669 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10b015a720 con 0x7f10b00c57e0 2024-03-28T15:01:44.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.405+0000 7f10a6ffd640 1 -- 172.21.15.186:0/3112961669 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f10a0007400 con 0x7f10b00c57e0 2024-03-28T15:01:44.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.405+0000 7f10a6ffd640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f109406c560 0x7f109406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.405+0000 7f10b6348640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f109406c560 0x7f109406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.405+0000 7f10a6ffd640 1 -- 172.21.15.186:0/3112961669 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f10a008ab00 con 0x7f10b00c57e0 2024-03-28T15:01:44.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.406+0000 7f10b6348640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f109406c560 0x7f109406ea20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f10ac036000 tx=0x7f10ac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:44.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.410+0000 7f10a6ffd640 1 -- 172.21.15.186:0/3112961669 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f10a0058a40 con 0x7f10b00c57e0 2024-03-28T15:01:44.588 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.587+0000 7f10b85ac640 1 -- 172.21.15.186:0/3112961669 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10b0069500 con 0x7f10b00c57e0 2024-03-28T15:01:44.590 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.588+0000 7f10a6ffd640 1 -- 172.21.15.186:0/3112961669 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f10a005cd40 con 0x7f10b00c57e0 2024-03-28T15:01:44.590 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:44.590 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:33.099254+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:44.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.590+0000 7f10b85ac640 1 -- 172.21.15.186:0/3112961669 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f109406c560 msgr2=0x7f109406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.590+0000 7f10b85ac640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f109406c560 0x7f109406ea20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f10ac036000 tx=0x7f10ac008030 comp rx=0 tx=0).stop 2024-03-28T15:01:44.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.590+0000 7f10b85ac640 1 -- 172.21.15.186:0/3112961669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b00c57e0 msgr2=0x7f10b0158b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.590+0000 7f10b85ac640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b00c57e0 0x7f10b0158b30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10a000c430 tx=0x7f10a000c830 comp rx=0 tx=0).stop 2024-03-28T15:01:44.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.590+0000 7f10b85ac640 1 -- 172.21.15.186:0/3112961669 shutdown_connections 2024-03-28T15:01:44.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.590+0000 7f10b85ac640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f109406c560 0x7f109406ea20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.590+0000 7f10b85ac640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10b0154bd0 0x7f10b01f3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.590+0000 7f10b85ac640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b00c57e0 0x7f10b0158b30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.592 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.590+0000 7f10b85ac640 1 --2- 172.21.15.186:0/3112961669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10b00c53c0 0x7f10b01585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.591+0000 7f10b85ac640 1 -- 172.21.15.186:0/3112961669 >> 172.21.15.186:0/3112961669 conn(0x7f10b00731f0 msgr2=0x7f10b00d00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:44.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.591+0000 7f10b85ac640 1 -- 172.21.15.186:0/3112961669 shutdown_connections 2024-03-28T15:01:44.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.591+0000 7f10b85ac640 1 -- 172.21.15.186:0/3112961669 wait complete. 2024-03-28T15:01:44.603 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:01:44.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.702+0000 7f03523f7640 1 Processor -- start 2024-03-28T15:01:44.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.702+0000 7f03523f7640 1 -- start start 2024-03-28T15:01:44.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.703+0000 7f03523f7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f034c0b1e30 0x7f034c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.703+0000 7f03523f7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f034c06caa0 con 0x7f034c151990 2024-03-28T15:01:44.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.703+0000 7f03523f7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f034c06cd80 con 0x7f034c0c2d00 2024-03-28T15:01:44.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.703+0000 7f03523f7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f034c0c1da0 con 0x7f034c0b1e30 2024-03-28T15:01:44.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.703+0000 7f034b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f034c0b1e30 0x7f034c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.703+0000 7f034b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f034c0b1e30 0x7f034c0712b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48642/0 (socket says 172.21.15.186:48642) 2024-03-28T15:01:44.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.703+0000 7f034b7fe640 1 -- 172.21.15.186:0/1721320876 learned_addr learned my addr 172.21.15.186:0/1721320876 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:44.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.703+0000 7f034b7fe640 1 -- 172.21.15.186:0/1721320876 >> v1:172.21.15.191:6789/0 conn(0x7f034c0c2d00 legacy=0x7f034c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.703+0000 7f034b7fe640 1 -- 172.21.15.186:0/1721320876 >> v1:172.21.15.186:6789/0 conn(0x7f034c151990 legacy=0x7f034c0c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.703+0000 7f034b7fe640 1 -- 172.21.15.186:0/1721320876 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c0c1340 con 0x7f034c0b1e30 2024-03-28T15:01:44.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.704+0000 7f034b7fe640 1 --2- 172.21.15.186:0/1721320876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f034c0b1e30 0x7f034c0712b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0340009870 tx=0x7f034002bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86335859113936a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.704+0000 7f034affd640 1 -- 172.21.15.186:0/1721320876 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034002d870 con 0x7f034c0b1e30 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.704+0000 7f034affd640 1 -- 172.21.15.186:0/1721320876 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f034002da10 con 0x7f034c0b1e30 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.704+0000 7f034affd640 1 -- 172.21.15.186:0/1721320876 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034002dde0 con 0x7f034c0b1e30 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.704+0000 7f03523f7640 1 -- 172.21.15.186:0/1721320876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f034c0b1e30 msgr2=0x7f034c0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.704+0000 7f03523f7640 1 --2- 172.21.15.186:0/1721320876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f034c0b1e30 0x7f034c0712b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0340009870 tx=0x7f034002bab0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.704+0000 7f03523f7640 1 -- 172.21.15.186:0/1721320876 shutdown_connections 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.704+0000 7f03523f7640 1 --2- 172.21.15.186:0/1721320876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f034c0b1e30 0x7f034c0712b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.704+0000 7f03523f7640 1 -- 172.21.15.186:0/1721320876 >> 172.21.15.186:0/1721320876 conn(0x7f034c0c8a20 msgr2=0x7f034c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.704+0000 7f03523f7640 1 -- 172.21.15.186:0/1721320876 shutdown_connections 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.704+0000 7f03523f7640 1 -- 172.21.15.186:0/1721320876 wait complete. 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.705+0000 7f03523f7640 1 Processor -- start 2024-03-28T15:01:44.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.706+0000 7f03523f7640 1 -- start start 2024-03-28T15:01:44.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.706+0000 7f03523f7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f034c0b1e30 0x7f034c14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.706+0000 7f03523f7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f034c0c2d00 0x7f034c1f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.706+0000 7f03523f7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f034c151990 0x7f034c1f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.706+0000 7f03523f7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f034c0c2940 con 0x7f034c0c2d00 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.706+0000 7f03523f7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f034c0c1900 con 0x7f034c0b1e30 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.706+0000 7f03523f7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f034c0c2ac0 con 0x7f034c151990 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.706+0000 7f034b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f034c0c2d00 0x7f034c1f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.706+0000 7f034b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f034c0c2d00 0x7f034c1f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48644/0 (socket says 172.21.15.186:48644) 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.706+0000 7f034b7fe640 1 -- 172.21.15.186:0/2031466885 learned_addr learned my addr 172.21.15.186:0/2031466885 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.706+0000 7f034bfff640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f034c0b1e30 0x7f034c14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f0350994640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f034c151990 0x7f034c1f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f034b7fe640 1 -- 172.21.15.186:0/2031466885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f034c151990 msgr2=0x7f034c1f49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f034b7fe640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f034c151990 0x7f034c1f49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f034b7fe640 1 -- 172.21.15.186:0/2031466885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f034c0b1e30 msgr2=0x7f034c14d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f034b7fe640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f034c0b1e30 0x7f034c14d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f034b7fe640 1 -- 172.21.15.186:0/2031466885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c1478b0 con 0x7f034c0c2d00 2024-03-28T15:01:44.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f034b7fe640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f034c0c2d00 0x7f034c1f4470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0340009990 tx=0x7f0340004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:44.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f0348ff9640 1 -- 172.21.15.186:0/2031466885 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340047070 con 0x7f034c0c2d00 2024-03-28T15:01:44.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f0348ff9640 1 -- 172.21.15.186:0/2031466885 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0340037a40 con 0x7f034c0c2d00 2024-03-28T15:01:44.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f03523f7640 1 -- 172.21.15.186:0/2031466885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034c1f11d0 con 0x7f034c0c2d00 2024-03-28T15:01:44.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f0348ff9640 1 -- 172.21.15.186:0/2031466885 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340037e10 con 0x7f034c0c2d00 2024-03-28T15:01:44.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.707+0000 7f03523f7640 1 -- 172.21.15.186:0/2031466885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034c1f1790 con 0x7f034c0c2d00 2024-03-28T15:01:44.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.708+0000 7f03523f7640 1 -- 172.21.15.186:0/2031466885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034c0b2210 con 0x7f034c0c2d00 2024-03-28T15:01:44.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.708+0000 7f0348ff9640 1 -- 172.21.15.186:0/2031466885 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f034004b3f0 con 0x7f034c0c2d00 2024-03-28T15:01:44.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.709+0000 7f0348ff9640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f033806c530 0x7f033806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:44.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.709+0000 7f034bfff640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f033806c530 0x7f033806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:44.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.709+0000 7f0348ff9640 1 -- 172.21.15.186:0/2031466885 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f03400ad2a0 con 0x7f034c0c2d00 2024-03-28T15:01:44.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.709+0000 7f034bfff640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f033806c530 0x7f033806e9f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f033c000f60 tx=0x7f033c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:44.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.714+0000 7f0348ff9640 1 -- 172.21.15.186:0/2031466885 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f034c0b2210 con 0x7f034c0c2d00 2024-03-28T15:01:44.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.927+0000 7f03523f7640 1 -- 172.21.15.186:0/2031466885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f034c14ba10 con 0x7f034c0c2d00 2024-03-28T15:01:44.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.927+0000 7f0348ff9640 1 -- 172.21.15.186:0/2031466885 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7f034c14ba10 con 0x7f034c0c2d00 2024-03-28T15:01:44.931 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:44.931 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:01:44.932 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:01:44.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.929+0000 7f03523f7640 1 -- 172.21.15.186:0/2031466885 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f033806c530 msgr2=0x7f033806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.929+0000 7f03523f7640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f033806c530 0x7f033806e9f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f033c000f60 tx=0x7f033c009040 comp rx=0 tx=0).stop 2024-03-28T15:01:44.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.929+0000 7f03523f7640 1 -- 172.21.15.186:0/2031466885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f034c0c2d00 msgr2=0x7f034c1f4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:44.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.929+0000 7f03523f7640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f034c0c2d00 0x7f034c1f4470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0340009990 tx=0x7f0340004730 comp rx=0 tx=0).stop 2024-03-28T15:01:44.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.929+0000 7f03523f7640 1 -- 172.21.15.186:0/2031466885 shutdown_connections 2024-03-28T15:01:44.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.929+0000 7f03523f7640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f033806c530 0x7f033806e9f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.929+0000 7f03523f7640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f034c151990 0x7f034c1f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.929+0000 7f03523f7640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f034c0c2d00 0x7f034c1f4470 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.929+0000 7f03523f7640 1 --2- 172.21.15.186:0/2031466885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f034c0b1e30 0x7f034c14d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:44.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.929+0000 7f03523f7640 1 -- 172.21.15.186:0/2031466885 >> 172.21.15.186:0/2031466885 conn(0x7f034c0c8a20 msgr2=0x7f034c0c7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:44.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.930+0000 7f03523f7640 1 -- 172.21.15.186:0/2031466885 shutdown_connections 2024-03-28T15:01:44.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:44.930+0000 7f03523f7640 1 -- 172.21.15.186:0/2031466885 wait complete. 2024-03-28T15:01:45.546 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T15:01:31.425299+0000', 'modified': '2024-03-28T15:01:38.125799+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4308}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4308': {'gid': 4308, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6837/2813406930', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6835', 'nonce': 2813406930}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 2813406930}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4308, 'qdb_cluster': [4308]} 2024-03-28T15:01:45.546 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T15:01:45.546 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-28T15:01:45.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.658+0000 7fc5127ee640 1 Processor -- start 2024-03-28T15:01:45.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.658+0000 7fc5127ee640 1 -- start start 2024-03-28T15:01:45.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc5127ee640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc50c0701e0 0x7fc50c0c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc5127ee640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc50c0c78b0 0x7fc50c15ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc5127ee640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc50c15ed90 con 0x7fc50c154c40 2024-03-28T15:01:45.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc5127ee640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc50c155020 con 0x7fc50c0701e0 2024-03-28T15:01:45.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc5127ee640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fc50c15e860 con 0x7fc50c0c78b0 2024-03-28T15:01:45.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc50b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc50c0701e0 0x7fc50c0c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc50b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc50c0701e0 0x7fc50c0c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48652/0 (socket says 172.21.15.186:48652) 2024-03-28T15:01:45.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc50b7fe640 1 -- 172.21.15.186:0/1410094837 learned_addr learned my addr 172.21.15.186:0/1410094837 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:45.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc50bfff640 1 --1- 172.21.15.186:0/1410094837 >> v1:172.21.15.186:6789/0 conn(0x7fc50c154c40 0x7fc50c0cc5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51460/0 (socket says 172.21.15.186:51460) 2024-03-28T15:01:45.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc510d8b640 1 --2- 172.21.15.186:0/1410094837 >> v2:172.21.15.191:3300/0 conn(0x7fc50c0c78b0 0x7fc50c15ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc50b7fe640 1 -- 172.21.15.186:0/1410094837 >> v2:172.21.15.191:3300/0 conn(0x7fc50c0c78b0 msgr2=0x7fc50c15ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:45.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/1410094837 >> v2:172.21.15.191:3300/0 conn(0x7fc50c0c78b0 0x7fc50c15ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc50b7fe640 1 -- 172.21.15.186:0/1410094837 >> v1:172.21.15.186:6789/0 conn(0x7fc50c154c40 legacy=0x7fc50c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.659+0000 7fc50b7fe640 1 -- 172.21.15.186:0/1410094837 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50c15a860 con 0x7fc50c0701e0 2024-03-28T15:01:45.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.660+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/1410094837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc50c0701e0 0x7fc50c0c8900 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0c77e0 tx=0x7fc50002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76efd55d1b840620 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:45.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.660+0000 7fc50affd640 1 -- 172.21.15.186:0/1410094837 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500035020 con 0x7fc50c0701e0 2024-03-28T15:01:45.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.660+0000 7fc50affd640 1 -- 172.21.15.186:0/1410094837 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc500031870 con 0x7fc50c0701e0 2024-03-28T15:01:45.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.660+0000 7fc50affd640 1 -- 172.21.15.186:0/1410094837 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500031c40 con 0x7fc50c0701e0 2024-03-28T15:01:45.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.660+0000 7fc5127ee640 1 -- 172.21.15.186:0/1410094837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc50c0701e0 msgr2=0x7fc50c0c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.660+0000 7fc5127ee640 1 --2- 172.21.15.186:0/1410094837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc50c0701e0 0x7fc50c0c8900 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0c77e0 tx=0x7fc50002b910 comp rx=0 tx=0).stop 2024-03-28T15:01:45.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.661+0000 7fc5127ee640 1 -- 172.21.15.186:0/1410094837 shutdown_connections 2024-03-28T15:01:45.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.661+0000 7fc5127ee640 1 --2- 172.21.15.186:0/1410094837 >> v2:172.21.15.191:3300/0 conn(0x7fc50c0c78b0 0x7fc50c15ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.661+0000 7fc5127ee640 1 --2- 172.21.15.186:0/1410094837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc50c0701e0 0x7fc50c0c8900 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.661+0000 7fc5127ee640 1 -- 172.21.15.186:0/1410094837 >> 172.21.15.186:0/1410094837 conn(0x7fc50c06df10 msgr2=0x7fc50c06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:45.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.661+0000 7fc5127ee640 1 -- 172.21.15.186:0/1410094837 shutdown_connections 2024-03-28T15:01:45.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.661+0000 7fc5127ee640 1 -- 172.21.15.186:0/1410094837 wait complete. 2024-03-28T15:01:45.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.662+0000 7fc5127ee640 1 Processor -- start 2024-03-28T15:01:45.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.662+0000 7fc5127ee640 1 -- start start 2024-03-28T15:01:45.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc5127ee640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc50c0701e0 0x7fc50c1f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc5127ee640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc50c0c78b0 0x7fc50c150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc5127ee640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc50c154c40 0x7fc50c1f0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc5127ee640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c0c7470 con 0x7fc50c0c78b0 2024-03-28T15:01:45.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc5127ee640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c0c7190 con 0x7fc50c0701e0 2024-03-28T15:01:45.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc5127ee640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc50c0ccd00 con 0x7fc50c154c40 2024-03-28T15:01:45.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc50b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc50c0c78b0 0x7fc50c150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.665 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc50b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc50c0c78b0 0x7fc50c150670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48662/0 (socket says 172.21.15.186:48662) 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc50b7fe640 1 -- 172.21.15.186:0/2273032749 learned_addr learned my addr 172.21.15.186:0/2273032749 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc50bfff640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc50c0701e0 0x7fc50c1f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc510d8b640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc50c154c40 0x7fc50c1f0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc50b7fe640 1 -- 172.21.15.186:0/2273032749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc50c154c40 msgr2=0x7fc50c1f0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc50c154c40 0x7fc50c1f0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc50b7fe640 1 -- 172.21.15.186:0/2273032749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc50c0701e0 msgr2=0x7fc50c1f42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc50c0701e0 0x7fc50c1f42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc50b7fe640 1 -- 172.21.15.186:0/2273032749 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50c1520d0 con 0x7fc50c0c78b0 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc50c0c78b0 0x7fc50c150670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc50002b950 tx=0x7fc5000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc508ff9640 1 -- 172.21.15.186:0/2273032749 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500040070 con 0x7fc50c0c78b0 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc508ff9640 1 -- 172.21.15.186:0/2273032749 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc500030d00 con 0x7fc50c0c78b0 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc5127ee640 1 -- 172.21.15.186:0/2273032749 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc50c0c6760 con 0x7fc50c0c78b0 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc508ff9640 1 -- 172.21.15.186:0/2273032749 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500035040 con 0x7fc50c0c78b0 2024-03-28T15:01:45.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.663+0000 7fc5127ee640 1 -- 172.21.15.186:0/2273032749 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc50c1f0350 con 0x7fc50c0c78b0 2024-03-28T15:01:45.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.665+0000 7fc508ff9640 1 -- 172.21.15.186:0/2273032749 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc50004b060 con 0x7fc50c0c78b0 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.665+0000 7fc508ff9640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc4f806c530 0x7fc4f806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.665+0000 7fc508ff9640 1 -- 172.21.15.186:0/2273032749 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc5000ad290 con 0x7fc50c0c78b0 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.665+0000 7fc50bfff640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc4f806c530 0x7fc4f806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.666+0000 7fc50bfff640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc4f806c530 0x7fc4f806e9f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc007fd0 tx=0x7fc4fc007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.666+0000 7fc5153cd740 1 Processor -- start 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.666+0000 7fc5153cd740 1 -- start start 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f45e60 0x55f7d3f40400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 0x55f7d3f6e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 0x55f7d3f6ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc5153cd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f7d3eaa010 con 0x55f7d3f46900 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc5153cd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f7d3e52770 con 0x55f7d3f45e60 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc5153cd740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x55f7d3ea5fc0 con 0x55f7d3f46240 2024-03-28T15:01:45.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc50b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 0x55f7d3f6e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc50b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 0x55f7d3f6e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48678/0 (socket says 172.21.15.186:48678) 2024-03-28T15:01:45.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc50b7fe640 1 -- 172.21.15.186:0/3445209196 learned_addr learned my addr 172.21.15.186:0/3445209196 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:45.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc50bfff640 1 --2- 172.21.15.186:0/3445209196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f45e60 0x55f7d3f40400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc510d8b640 1 --2- 172.21.15.186:0/3445209196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 0x55f7d3f6ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc50b7fe640 1 -- 172.21.15.186:0/3445209196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 msgr2=0x55f7d3f6ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:45.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/3445209196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 0x55f7d3f6ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc50b7fe640 1 -- 172.21.15.186:0/3445209196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f45e60 msgr2=0x55f7d3f40400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:45.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/3445209196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f45e60 0x55f7d3f40400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.667+0000 7fc50b7fe640 1 -- 172.21.15.186:0/3445209196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7d3e71330 con 0x55f7d3f46900 2024-03-28T15:01:45.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/3445209196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 0x55f7d3f6e820 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc5000ab730 tx=0x7fc5000094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ed7a7fc457b4e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc4f1ffb640 1 -- 172.21.15.186:0/3445209196 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5000a7070 con 0x55f7d3f46900 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc4f1ffb640 1 -- 172.21.15.186:0/3445209196 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5000ad0b0 con 0x55f7d3f46900 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc4f1ffb640 1 -- 172.21.15.186:0/3445209196 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50007b260 con 0x55f7d3f46900 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc5153cd740 1 -- 172.21.15.186:0/3445209196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 msgr2=0x55f7d3f6e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc5153cd740 1 --2- 172.21.15.186:0/3445209196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 0x55f7d3f6e820 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc5000ab730 tx=0x7fc5000094d0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc5153cd740 1 -- 172.21.15.186:0/3445209196 shutdown_connections 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc5153cd740 1 --2- 172.21.15.186:0/3445209196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 0x55f7d3f6ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc5153cd740 1 --2- 172.21.15.186:0/3445209196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 0x55f7d3f6e820 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc5153cd740 1 --2- 172.21.15.186:0/3445209196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f45e60 0x55f7d3f40400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc5153cd740 1 -- 172.21.15.186:0/3445209196 >> 172.21.15.186:0/3445209196 conn(0x55f7d3f0cb20 msgr2=0x55f7d3f44f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc5153cd740 1 -- 172.21.15.186:0/3445209196 shutdown_connections 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.668+0000 7fc5153cd740 1 -- 172.21.15.186:0/3445209196 wait complete. 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.669+0000 7fc5153cd740 1 -- start start 2024-03-28T15:01:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.669+0000 7fc5153cd740 1 Processor -- start 2024-03-28T15:01:45.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f45e60 0x55f7d3f66320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f46240 0x55f7d3f83450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46900 0x55f7d3f873f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc5153cd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f7d3eaa010 con 0x55f7d3f45e60 2024-03-28T15:01:45.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc5153cd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f7d3e65b30 con 0x55f7d3f46240 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc5153cd740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x55f7d3f84ec0 con 0x55f7d3f46900 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc5153cd740 10 client.0 _ll_register_callbacks cb 0x55f7d3ea11a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc50bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f45e60 0x55f7d3f66320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc50bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f45e60 0x55f7d3f66320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48684/0 (socket says 172.21.15.186:48684) 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc50bfff640 1 -- 172.21.15.186:0/609099651 learned_addr learned my addr 172.21.15.186:0/609099651 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f46240 0x55f7d3f83450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc510d8b640 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46900 0x55f7d3f873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc50bfff640 1 -- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46900 msgr2=0x55f7d3f873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc50bfff640 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46900 0x55f7d3f873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc50bfff640 1 -- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f46240 msgr2=0x55f7d3f83450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc50bfff640 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f46240 0x55f7d3f83450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc50bfff640 1 -- 172.21.15.186:0/609099651 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7d3bbe2d0 con 0x55f7d3f45e60 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc50bfff640 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f45e60 0x55f7d3f66320 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0136a0 tx=0x7fc4fc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5fc811431136ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.670+0000 7fc4eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc4eaffd640 1 -- 172.21.15.186:0/609099651 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc020640 con 0x55f7d3f45e60 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc4eaffd640 1 -- 172.21.15.186:0/609099651 <== mon.0 v2:172.21.15.186:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc4fc0207e0 con 0x55f7d3f45e60 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc5153cd740 1 -- 172.21.15.186:0/609099651 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5000ad0b0 con 0x55f7d3f45e60 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc4eaffd640 1 -- 172.21.15.186:0/609099651 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fc4fc02e050 con 0x55f7d3f45e60 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc5153cd740 10 client.4680 fetch_fsmap learned FSMap version 8 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc5153cd740 1 -- 172.21.15.186:0/609099651 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55f7d3c886b0 con 0x55f7d3f45e60 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc4eaffd640 1 -- 172.21.15.186:0/609099651 <== mon.0 v2:172.21.15.186:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5000ad0b0 con 0x55f7d3f45e60 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc4eaffd640 1 -- 172.21.15.186:0/609099651 <== mon.0 v2:172.21.15.186:3300/0 5 ==== fsmap(e 8) ==== 4704+0+0 (secure 0 0 0) 0x7fc4fc031020 con 0x55f7d3f45e60 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc5153cd740 10 client.4680 fetch_fsmap finished waiting for FSMap version 8 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc5153cd740 10 client.4680 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc5153cd740 20 client.4680 populate_metadata read hostname 'smithi186' 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc5153cd740 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:6835/2813406930,v1:172.21.15.191:6837/2813406930] conn(0x55f7d3f82200 0x55f7d3fa50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc5153cd740 4 client.4680 mds_command: new command op to 4308 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc5153cd740 1 -- 172.21.15.186:0/609099651 --> [v2:172.21.15.191:6835/2813406930,v1:172.21.15.191:6837/2813406930] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7d3eb7550 con 0x55f7d3f82200 2024-03-28T15:01:45.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.671+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:6835/2813406930,v1:172.21.15.191:6837/2813406930] conn(0x55f7d3f82200 0x55f7d3fa50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.672+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:6835/2813406930,v1:172.21.15.191:6837/2813406930] conn(0x55f7d3f82200 0x55f7d3fa50e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73aa06d670a77c64 server_cookie=78b331d2964ce266 in_seq=0 out_seq=0 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.672+0000 7fc4eaffd640 10 client.4680 ms_handle_connect on v2:172.21.15.191:6835/2813406930 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.672+0000 7fc4eaffd640 1 -- 172.21.15.186:0/609099651 <== mds.0 v2:172.21.15.191:6835/2813406930 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7d3eb7550 con 0x55f7d3f82200 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.672+0000 7fc4eaffd640 10 client.4680 handle_command_reply: tid=0 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.672+0000 7fc5153cd740 1 client.4680 shutdown 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.672+0000 7fc5153cd740 1 -- 172.21.15.186:0/609099651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f45e60 msgr2=0x55f7d3f66320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.672+0000 7fc5153cd740 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f45e60 0x55f7d3f66320 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0136a0 tx=0x7fc4fc000bc0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.672+0000 7fc5153cd740 1 -- 172.21.15.186:0/609099651 shutdown_connections 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.672+0000 7fc5153cd740 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:6835/2813406930,v1:172.21.15.191:6837/2813406930] conn(0x55f7d3f82200 0x55f7d3fa50e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.673+0000 7fc5153cd740 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46900 0x55f7d3f873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.673+0000 7fc5153cd740 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f46240 0x55f7d3f83450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.673+0000 7fc5153cd740 1 --2- 172.21.15.186:0/609099651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f45e60 0x55f7d3f66320 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.673+0000 7fc4eaffd640 0 client.4680 ms_handle_reset on v2:172.21.15.191:6835/2813406930 2024-03-28T15:01:45.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.673+0000 7fc5153cd740 1 -- 172.21.15.186:0/609099651 >> 172.21.15.186:0/609099651 conn(0x55f7d3f0cb20 msgr2=0x55f7d3f3f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:45.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.673+0000 7fc5153cd740 1 -- 172.21.15.186:0/609099651 shutdown_connections 2024-03-28T15:01:45.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.673+0000 7fc5153cd740 1 -- 172.21.15.186:0/609099651 wait complete. 2024-03-28T15:01:45.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.673+0000 7fc5153cd740 20 client.4680 trim_cache size 0 max 16384 2024-03-28T15:01:45.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.691+0000 7fc5153cd740 1 Processor -- start 2024-03-28T15:01:45.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.691+0000 7fc5153cd740 1 -- start start 2024-03-28T15:01:45.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f45e60 0x55f7d3f863d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46240 0x55f7d3f402a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f46900 0x55f7d3f407e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc5153cd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c15dac0 con 0x55f7d3f46240 2024-03-28T15:01:45.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc5153cd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f7d3f83ba0 con 0x55f7d3f46900 2024-03-28T15:01:45.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc5153cd740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x55f7d3f81100 con 0x55f7d3f45e60 2024-03-28T15:01:45.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc50b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46240 0x55f7d3f402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc50b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46240 0x55f7d3f402a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48688/0 (socket says 172.21.15.186:48688) 2024-03-28T15:01:45.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc50b7fe640 1 -- 172.21.15.186:0/3765612667 learned_addr learned my addr 172.21.15.186:0/3765612667 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:45.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc50bfff640 1 --2- 172.21.15.186:0/3765612667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f45e60 0x55f7d3f863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc510d8b640 1 --2- 172.21.15.186:0/3765612667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f46900 0x55f7d3f407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc50b7fe640 1 -- 172.21.15.186:0/3765612667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f45e60 msgr2=0x55f7d3f863d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/3765612667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f45e60 0x55f7d3f863d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc50b7fe640 1 -- 172.21.15.186:0/3765612667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f46900 msgr2=0x55f7d3f407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/3765612667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f46900 0x55f7d3f407e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc50b7fe640 1 -- 172.21.15.186:0/3765612667 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7d3f84ac0 con 0x55f7d3f46240 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/3765612667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46240 0x55f7d3f402a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc5000ad5b0 tx=0x7fc50007e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9192a01b9c52bb50 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc4f17fa640 1 -- 172.21.15.186:0/3765612667 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5000a7040 con 0x55f7d3f46240 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc4f17fa640 1 -- 172.21.15.186:0/3765612667 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc500083d90 con 0x55f7d3f46240 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.692+0000 7fc4f17fa640 1 -- 172.21.15.186:0/3765612667 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50007b260 con 0x55f7d3f46240 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.693+0000 7fc5153cd740 1 -- 172.21.15.186:0/3765612667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46240 msgr2=0x55f7d3f402a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.693+0000 7fc5153cd740 1 --2- 172.21.15.186:0/3765612667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46240 0x55f7d3f402a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc5000ad5b0 tx=0x7fc50007e000 comp rx=0 tx=0).stop 2024-03-28T15:01:45.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.693+0000 7fc5153cd740 1 -- 172.21.15.186:0/3765612667 shutdown_connections 2024-03-28T15:01:45.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.693+0000 7fc5153cd740 1 --2- 172.21.15.186:0/3765612667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f46900 0x55f7d3f407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.693+0000 7fc5153cd740 1 --2- 172.21.15.186:0/3765612667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46240 0x55f7d3f402a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.693+0000 7fc5153cd740 1 --2- 172.21.15.186:0/3765612667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f45e60 0x55f7d3f863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.693+0000 7fc5153cd740 1 -- 172.21.15.186:0/3765612667 >> 172.21.15.186:0/3765612667 conn(0x55f7d3f0cb20 msgr2=0x55f7d3f82850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:45.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.693+0000 7fc5153cd740 1 -- 172.21.15.186:0/3765612667 shutdown_connections 2024-03-28T15:01:45.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.693+0000 7fc5153cd740 1 -- 172.21.15.186:0/3765612667 wait complete. 2024-03-28T15:01:45.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.693+0000 7fc5153cd740 1 -- start start 2024-03-28T15:01:45.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc5153cd740 1 Processor -- start 2024-03-28T15:01:45.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f45e60 0x55f7d3f45290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 0x55f7d3faa700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc5153cd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 0x55f7d3ffa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc5153cd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f7d3f826d0 con 0x55f7d3f46900 2024-03-28T15:01:45.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc5153cd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f7d3f82080 con 0x55f7d3f45e60 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc5153cd740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x55f7d3f966a0 con 0x55f7d3f46240 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc5153cd740 10 client.0 _ll_register_callbacks cb 0x7fc4fc004b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc50b7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 0x55f7d3faa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc510d8b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 0x55f7d3ffa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc50b7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 0x55f7d3faa700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:52008/0 (socket says 172.21.15.186:52008) 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc50b7fe640 1 -- 172.21.15.186:0/951269759 learned_addr learned my addr 172.21.15.186:0/951269759 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.694+0000 7fc50bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f45e60 0x55f7d3f45290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc510d8b640 1 -- 172.21.15.186:0/951269759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 msgr2=0x55f7d3faa700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc510d8b640 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 0x55f7d3faa700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc510d8b640 1 -- 172.21.15.186:0/951269759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f45e60 msgr2=0x55f7d3f45290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc510d8b640 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f45e60 0x55f7d3f45290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc510d8b640 1 -- 172.21.15.186:0/951269759 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7d3f8ab80 con 0x55f7d3f46900 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc50b7fe640 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 0x55f7d3faa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:45.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc510d8b640 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 0x55f7d3ffa5f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55f7d3f8ada0 tx=0x7fc5040187f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4cbf4dc014411e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc4eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc4eaffd640 1 -- 172.21.15.186:0/951269759 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504003990 con 0x55f7d3f46900 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc4eaffd640 1 -- 172.21.15.186:0/951269759 <== mon.0 v2:172.21.15.186:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc504003b30 con 0x55f7d3f46900 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc5153cd740 1 -- 172.21.15.186:0/951269759 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc500083d90 con 0x55f7d3f46900 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc4eaffd640 1 -- 172.21.15.186:0/951269759 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fc50402a050 con 0x55f7d3f46900 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc5153cd740 10 client.4686 fetch_fsmap learned FSMap version 8 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc5153cd740 1 -- 172.21.15.186:0/951269759 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55f7d4000d50 con 0x55f7d3f46900 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc4eaffd640 1 -- 172.21.15.186:0/951269759 <== mon.0 v2:172.21.15.186:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc500083d90 con 0x55f7d3f46900 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc4eaffd640 1 -- 172.21.15.186:0/951269759 <== mon.0 v2:172.21.15.186:3300/0 5 ==== fsmap(e 8) ==== 4704+0+0 (secure 0 0 0) 0x7fc50402d020 con 0x55f7d3f46900 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc5153cd740 10 client.4686 fetch_fsmap finished waiting for FSMap version 8 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc5153cd740 10 client.4686 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T15:01:45.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc5153cd740 20 client.4686 populate_metadata read hostname 'smithi186' 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc5153cd740 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.191:6835/2813406930,v1:172.21.15.191:6837/2813406930] conn(0x55f7d3f3fca0 0x55f7d400d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc5153cd740 4 client.4686 mds_command: new command op to 4308 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.695+0000 7fc5153cd740 1 -- 172.21.15.186:0/951269759 --> [v2:172.21.15.191:6835/2813406930,v1:172.21.15.191:6837/2813406930] -- command(tid 0: {"prefix": "status"}) -- 0x55f7d3eb18a0 con 0x55f7d3f3fca0 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc50bfff640 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.191:6835/2813406930,v1:172.21.15.191:6837/2813406930] conn(0x55f7d3f3fca0 0x55f7d400d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc4eaffd640 10 client.4686 ms_handle_connect on v2:172.21.15.191:6835/2813406930 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc50bfff640 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.191:6835/2813406930,v1:172.21.15.191:6837/2813406930] conn(0x55f7d3f3fca0 0x55f7d400d1f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48e776f884cadeb6 server_cookie=fd6f21b92066059 in_seq=0 out_seq=0 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc4eaffd640 1 -- 172.21.15.186:0/951269759 <== mds.0 v2:172.21.15.191:6835/2813406930 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x55f7d3eb18a0 con 0x55f7d3f3fca0 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc4eaffd640 10 client.4686 handle_command_reply: tid=0 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 1 client.4686 shutdown 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 1 -- 172.21.15.186:0/951269759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 msgr2=0x55f7d3ffa5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 0x55f7d3ffa5f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55f7d3f8ada0 tx=0x7fc5040187f0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 1 -- 172.21.15.186:0/951269759 shutdown_connections 2024-03-28T15:01:45.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.191:6835/2813406930,v1:172.21.15.191:6837/2813406930] conn(0x55f7d3f3fca0 0x55f7d400d1f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d3f46900 0x55f7d3ffa5f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55f7d3f46240 0x55f7d3faa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 1 --2- 172.21.15.186:0/951269759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f7d3f45e60 0x55f7d3f45290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 1 -- 172.21.15.186:0/951269759 >> 172.21.15.186:0/951269759 conn(0x55f7d3f0cb20 msgr2=0x55f7d3f863d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:45.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc4eaffd640 0 client.4686 ms_handle_reset on v2:172.21.15.191:6835/2813406930 2024-03-28T15:01:45.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 1 -- 172.21.15.186:0/951269759 shutdown_connections 2024-03-28T15:01:45.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 1 -- 172.21.15.186:0/951269759 wait complete. 2024-03-28T15:01:45.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.696+0000 7fc5153cd740 20 client.4686 trim_cache size 0 max 16384 2024-03-28T15:01:45.704 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout: "cluster_fsid": "694dd083-ff98-42dc-bf10-681382259053", 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout: "whoami": 0, 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 4308, 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout: "want_state": "up:active", 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout: "state": "up:active", 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout: "fs_name": "cephfs", 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout: "rank_uptime": 13.584817998, 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout: "mdsmap_epoch": 8, 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_epoch": 20, 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_epoch_barrier": 20, 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout: "uptime": 20.627818584 2024-03-28T15:01:45.705 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-28T15:01:45.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 -- 172.21.15.186:0/2273032749 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc4f806c530 msgr2=0x7fc4f806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc4f806c530 0x7fc4f806e9f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc007fd0 tx=0x7fc4fc007e80 comp rx=0 tx=0).stop 2024-03-28T15:01:45.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 -- 172.21.15.186:0/2273032749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc50c0c78b0 msgr2=0x7fc50c150670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc50c0c78b0 0x7fc50c150670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc50002b950 tx=0x7fc5000046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 -- 172.21.15.186:0/2273032749 shutdown_connections 2024-03-28T15:01:45.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc4f806c530 0x7fc4f806e9f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc50c154c40 0x7fc50c1f0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc50c0c78b0 0x7fc50c150670 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 --2- 172.21.15.186:0/2273032749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc50c0701e0 0x7fc50c1f42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 -- 172.21.15.186:0/2273032749 >> 172.21.15.186:0/2273032749 conn(0x7fc50c06df10 msgr2=0x7fc50c071b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:45.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 -- 172.21.15.186:0/2273032749 shutdown_connections 2024-03-28T15:01:45.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.697+0000 7fc5127ee640 1 -- 172.21.15.186:0/2273032749 wait complete. 2024-03-28T15:01:45.708 INFO:teuthology.run_tasks:Running task sequential... 2024-03-28T15:01:45.717 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-28T15:01:45.717 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-28T15:01:45.718 INFO:teuthology.task.print:Enabling mgr modules 2024-03-28T15:01:45.718 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-28T15:01:45.760 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-28T15:01:45.760 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-28T15:01:45.760 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-28T15:01:45.760 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T15:01:45.760 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T15:01:45.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:45.761 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2024-03-28T15:01:45.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:45.777 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link delete ceph-brx 2024-03-28T15:01:45.846 INFO:teuthology.orchestra.run.smithi186.stderr:Cannot find device "ceph-brx" 2024-03-28T15:01:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:01:45.848 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-28T15:01:45.848 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-28T15:01:45.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.988+0000 7f11ac246640 1 Processor -- start 2024-03-28T15:01:45.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.989+0000 7f11ac246640 1 -- start start 2024-03-28T15:01:45.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.989+0000 7f11ac246640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f11a40caa70 0x7f11a40c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.989+0000 7f11ac246640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11a40c5380 0x7f11a40cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.989+0000 7f11ac246640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a40c6f90 con 0x7f11a40c6a00 2024-03-28T15:01:45.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.989+0000 7f11ac246640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f11a4071a80 con 0x7f11a40caa70 2024-03-28T15:01:45.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.989+0000 7f11ac246640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f11a40c6030 con 0x7f11a40c5380 2024-03-28T15:01:45.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.989+0000 7f11a9fe2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f11a40caa70 0x7f11a40c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.989+0000 7f11a9fe2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f11a40caa70 0x7f11a40c64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48710/0 (socket says 172.21.15.186:48710) 2024-03-28T15:01:45.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.989+0000 7f11a9fe2640 1 -- 172.21.15.186:0/25527592 learned_addr learned my addr 172.21.15.186:0/25527592 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:45.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.989+0000 7f11aa7e3640 1 --2- 172.21.15.186:0/25527592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11a40c5380 0x7f11a40cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.990+0000 7f11a9fe2640 1 -- 172.21.15.186:0/25527592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11a40c5380 msgr2=0x7f11a40cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:45.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.990+0000 7f11a9fe2640 1 --2- 172.21.15.186:0/25527592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11a40c5380 0x7f11a40cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.990+0000 7f11a9fe2640 1 -- 172.21.15.186:0/25527592 >> v1:172.21.15.191:6789/0 conn(0x7f11a40c6a00 legacy=0x7f11a40cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.990+0000 7f11a9fe2640 1 -- 172.21.15.186:0/25527592 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a40cc970 con 0x7f11a40caa70 2024-03-28T15:01:45.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.990+0000 7f11a9fe2640 1 --2- 172.21.15.186:0/25527592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11a40caa70 0x7f11a40c64c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f119c009870 tx=0x7f119c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28812d53cfc7d21 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:45.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.991+0000 7f11a8fe0640 1 -- 172.21.15.186:0/25527592 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119c034070 con 0x7f11a40caa70 2024-03-28T15:01:45.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.991+0000 7f11a8fe0640 1 -- 172.21.15.186:0/25527592 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f119c030840 con 0x7f11a40caa70 2024-03-28T15:01:45.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.991+0000 7f11a8fe0640 1 -- 172.21.15.186:0/25527592 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119c030c10 con 0x7f11a40caa70 2024-03-28T15:01:45.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.991+0000 7f11ac246640 1 -- 172.21.15.186:0/25527592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11a40caa70 msgr2=0x7f11a40c64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.991+0000 7f11ac246640 1 --2- 172.21.15.186:0/25527592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11a40caa70 0x7f11a40c64c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f119c009870 tx=0x7f119c02b980 comp rx=0 tx=0).stop 2024-03-28T15:01:45.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.991+0000 7f11ac246640 1 -- 172.21.15.186:0/25527592 shutdown_connections 2024-03-28T15:01:45.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.991+0000 7f11ac246640 1 --2- 172.21.15.186:0/25527592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11a40c5380 0x7f11a40cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.991+0000 7f11ac246640 1 --2- 172.21.15.186:0/25527592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11a40caa70 0x7f11a40c64c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.991+0000 7f11ac246640 1 -- 172.21.15.186:0/25527592 >> 172.21.15.186:0/25527592 conn(0x7f11a40d00f0 msgr2=0x7f11a40d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:45.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.991+0000 7f11ac246640 1 -- 172.21.15.186:0/25527592 shutdown_connections 2024-03-28T15:01:45.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.991+0000 7f11ac246640 1 -- 172.21.15.186:0/25527592 wait complete. 2024-03-28T15:01:45.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.992+0000 7f11ac246640 1 Processor -- start 2024-03-28T15:01:45.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.992+0000 7f11ac246640 1 -- start start 2024-03-28T15:01:45.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11ac246640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11a40c5380 0x7f11a40b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11ac246640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11a40c6a00 0x7f11a40b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11ac246640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11a40caa70 0x7f11a414fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11ac246640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f11a40c5cc0 con 0x7f11a40c6a00 2024-03-28T15:01:45.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11ac246640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f11a415b530 con 0x7f11a40caa70 2024-03-28T15:01:45.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11ac246640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f11a415b810 con 0x7f11a40c5380 2024-03-28T15:01:45.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11a97e1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11a40c6a00 0x7f11a40b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11a97e1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11a40c6a00 0x7f11a40b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48720/0 (socket says 172.21.15.186:48720) 2024-03-28T15:01:45.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11a97e1640 1 -- 172.21.15.186:0/8136397 learned_addr learned my addr 172.21.15.186:0/8136397 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11aa7e3640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11a40caa70 0x7f11a414fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11a9fe2640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11a40c5380 0x7f11a40b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11a97e1640 1 -- 172.21.15.186:0/8136397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11a40c5380 msgr2=0x7f11a40b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11a97e1640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11a40c5380 0x7f11a40b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11a97e1640 1 -- 172.21.15.186:0/8136397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11a40caa70 msgr2=0x7f11a414fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11a97e1640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11a40caa70 0x7f11a414fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.993+0000 7f11a97e1640 1 -- 172.21.15.186:0/8136397 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a4159450 con 0x7f11a40c6a00 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.994+0000 7f11a97e1640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11a40c6a00 0x7f11a40b51b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f119400c470 tx=0x7f1194001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.994+0000 7f119a7fc640 1 -- 172.21.15.186:0/8136397 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11940052d0 con 0x7f11a40c6a00 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.994+0000 7f119a7fc640 1 -- 172.21.15.186:0/8136397 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1194005470 con 0x7f11a40c6a00 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.994+0000 7f11ac246640 1 -- 172.21.15.186:0/8136397 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a4159730 con 0x7f11a40c6a00 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.994+0000 7f119a7fc640 1 -- 172.21.15.186:0/8136397 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1194007260 con 0x7f11a40c6a00 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.994+0000 7f11ac246640 1 -- 172.21.15.186:0/8136397 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a4159cf0 con 0x7f11a40c6a00 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.995+0000 7f11ac246640 1 -- 172.21.15.186:0/8136397 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a40686d0 con 0x7f11a40c6a00 2024-03-28T15:01:45.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.995+0000 7f119a7fc640 1 -- 172.21.15.186:0/8136397 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1194007400 con 0x7f11a40c6a00 2024-03-28T15:01:45.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.996+0000 7f119a7fc640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f118806c560 0x7f118806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:45.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.996+0000 7f11a9fe2640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f118806c560 0x7f118806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:45.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.996+0000 7f119a7fc640 1 -- 172.21.15.186:0/8136397 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f119408aad0 con 0x7f11a40c6a00 2024-03-28T15:01:45.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:45.996+0000 7f11a9fe2640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f118806c560 0x7f118806ea20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f119c036000 tx=0x7f119c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:46.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.000+0000 7f119a7fc640 1 -- 172.21.15.186:0/8136397 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f11a40c6f90 con 0x7f11a40c6a00 2024-03-28T15:01:46.064 INFO:tasks.ceph.mds.f.smithi191.stderr:2024-03-28T15:01:46.065+0000 7ff929caa640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T15:01:46.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.214+0000 7f11ac246640 1 -- 172.21.15.186:0/8136397 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f11a40b3b10 con 0x7f11a40c6a00 2024-03-28T15:01:46.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.215+0000 7f119a7fc640 1 -- 172.21.15.186:0/8136397 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f1194058b30 con 0x7f11a40c6a00 2024-03-28T15:01:46.215 INFO:teuthology.orchestra.run.smithi186.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-28T15:01:46.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.216+0000 7f11ac246640 1 -- 172.21.15.186:0/8136397 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f118806c560 msgr2=0x7f118806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:46.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.216+0000 7f11ac246640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f118806c560 0x7f118806ea20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f119c036000 tx=0x7f119c03b040 comp rx=0 tx=0).stop 2024-03-28T15:01:46.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.216+0000 7f11ac246640 1 -- 172.21.15.186:0/8136397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11a40c6a00 msgr2=0x7f11a40b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:46.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.216+0000 7f11ac246640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11a40c6a00 0x7f11a40b51b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f119400c470 tx=0x7f1194001e70 comp rx=0 tx=0).stop 2024-03-28T15:01:46.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.217+0000 7f11ac246640 1 -- 172.21.15.186:0/8136397 shutdown_connections 2024-03-28T15:01:46.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.217+0000 7f11ac246640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f118806c560 0x7f118806ea20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:46.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.217+0000 7f11ac246640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11a40caa70 0x7f11a414fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:46.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.217+0000 7f11ac246640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11a40c6a00 0x7f11a40b51b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:46.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.217+0000 7f11ac246640 1 --2- 172.21.15.186:0/8136397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11a40c5380 0x7f11a40b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:46.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.217+0000 7f11ac246640 1 -- 172.21.15.186:0/8136397 >> 172.21.15.186:0/8136397 conn(0x7f11a40d00f0 msgr2=0x7f11a4154220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:46.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.217+0000 7f11ac246640 1 -- 172.21.15.186:0/8136397 shutdown_connections 2024-03-28T15:01:46.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46.217+0000 7f11ac246640 1 -- 172.21.15.186:0/8136397 wait complete. 2024-03-28T15:01:46.228 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T15:01:46.228 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T15:01:46.228 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi186.front.sepia.ceph.com 2024-03-28T15:01:46.228 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T15:01:46.228 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T15:01:46.228 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T15:01:46.228 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T15:01:46.228 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T15:01:46.228 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T15:01:46.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:46.229 DEBUG:teuthology.orchestra.run.smithi186:> ip addr 2024-03-28T15:01:46.244 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T15:01:46.244 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T15:01:46.244 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T15:01:46.244 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-03-28T15:01:46.244 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2024-03-28T15:01:46.244 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-03-28T15:01:46.244 INFO:teuthology.orchestra.run.smithi186.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T15:01:46.244 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2c brd ff:ff:ff:ff:ff:ff 2024-03-28T15:01:46.244 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f0 2024-03-28T15:01:46.244 INFO:teuthology.orchestra.run.smithi186.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T15:01:46.244 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2d brd ff:ff:ff:ff:ff:ff 2024-03-28T15:01:46.245 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f1 2024-03-28T15:01:46.245 INFO:teuthology.orchestra.run.smithi186.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T15:01:46.245 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f4 brd ff:ff:ff:ff:ff:ff 2024-03-28T15:01:46.245 INFO:teuthology.orchestra.run.smithi186.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T15:01:46.245 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f5 brd ff:ff:ff:ff:ff:ff 2024-03-28T15:01:46.245 INFO:teuthology.orchestra.run.smithi186.stdout: inet 172.21.15.186/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T15:01:46.245 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft 42688sec preferred_lft 42688sec 2024-03-28T15:01:46.245 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::ec4:7aff:fe88:6df5/64 scope link 2024-03-28T15:01:46.245 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2024-03-28T15:01:46.246 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T15:01:46.246 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T15:01:46.246 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-03-28T15:01:46.246 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link add name ceph-brx type bridge 2024-03-28T15:01:46.246 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip addr flush dev ceph-brx 2024-03-28T15:01:46.246 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set ceph-brx up 2024-03-28T15:01:46.246 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T15:01:46.246 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-03-28T15:01:46.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T15:01:46.427 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T15:01:46.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:46.433 DEBUG:teuthology.orchestra.run.smithi186:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T15:01:46.461 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-03-28T15:01:46.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:46.462 DEBUG:teuthology.orchestra.run.smithi186:> route 2024-03-28T15:01:46.534 INFO:teuthology.orchestra.run.smithi186.stdout:Kernel IP routing table 2024-03-28T15:01:46.534 INFO:teuthology.orchestra.run.smithi186.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T15:01:46.535 INFO:teuthology.orchestra.run.smithi186.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T15:01:46.535 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T15:01:46.535 INFO:teuthology.orchestra.run.smithi186.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T15:01:46.536 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T15:01:46.536 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-03-28T15:01:46.536 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T15:01:46.536 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T15:01:46.536 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T15:01:46.536 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-03-28T15:01:46.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T15:01:46.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T15:01:46.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:46.881 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2024-03-28T15:01:46.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:46.897 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list-id 2024-03-28T15:01:46.955 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T15:01:46.955 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-03-28T15:01:46.955 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T15:01:46.955 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T15:01:46.955 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-03-28T15:01:47.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T15:01:47.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T15:01:47.079 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T15:01:47.079 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T15:01:47.079 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-03-28T15:01:47.079 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T15:01:47.080 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T15:01:47.080 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T15:01:47.080 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T15:01:47.080 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T15:01:47.080 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-03-28T15:01:47.129 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T15:01:47.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T15:01:47.223 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T15:01:47.223 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2024-03-28T15:01:47.223 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set brx.0 up 2024-03-28T15:01:47.223 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T15:01:47.223 DEBUG:teuthology.orchestra.run.smithi186:> ') 2024-03-28T15:01:47.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T15:01:47.300 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T15:01:47.306 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-28T15:01:47.307 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T15:01:47.307 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T15:01:47.323 INFO:teuthology.orchestra.run.smithi186.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T15:01:47.324 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T15:01:47.324 DEBUG:teuthology.orchestra.run.smithi186:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T15:01:47.379 DEBUG:teuthology.orchestra.run.smithi186:> sudo modprobe fuse 2024-03-28T15:01:47.449 DEBUG:teuthology.orchestra.run.smithi186:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T15:01:47.508 INFO:teuthology.orchestra.run.smithi186.stdout:/proc 2024-03-28T15:01:47.508 INFO:teuthology.orchestra.run.smithi186.stdout:/sys 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/dev 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/kernel/security 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/shm 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/pts 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/run 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/fs/cgroup 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/fs/pstore 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/fs/bpf 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/ 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/fs/selinux 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/mqueue 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/hugepages 2024-03-28T15:01:47.509 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/kernel/debug 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/kernel/tracing 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/fs/fuse/connections 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/kernel/config 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/run/user/1000 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/ceph 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/run/netns 2024-03-28T15:01:47.510 INFO:teuthology.orchestra.run.smithi186.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T15:01:47.511 INFO:teuthology.orchestra.run.smithi186.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T15:01:47.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:47.511 DEBUG:teuthology.orchestra.run.smithi186:> ls /sys/fs/fuse/connections 2024-03-28T15:01:47.569 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T15:01:47.569 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-28T15:01:47.615 DEBUG:teuthology.orchestra.run.smithi186:> sudo modprobe fuse 2024-03-28T15:01:47.643 DEBUG:teuthology.orchestra.run.smithi186:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T15:01:47.701 INFO:teuthology.orchestra.run.smithi186.stdout:/proc 2024-03-28T15:01:47.701 INFO:teuthology.orchestra.run.smithi186.stdout:/sys 2024-03-28T15:01:47.701 INFO:teuthology.orchestra.run.smithi186.stdout:/dev 2024-03-28T15:01:47.701 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/kernel/security 2024-03-28T15:01:47.701 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/shm 2024-03-28T15:01:47.701 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/pts 2024-03-28T15:01:47.701 INFO:teuthology.orchestra.run.smithi186.stdout:/run 2024-03-28T15:01:47.702 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/fs/cgroup 2024-03-28T15:01:47.702 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/fs/pstore 2024-03-28T15:01:47.702 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/fs/bpf 2024-03-28T15:01:47.702 INFO:teuthology.orchestra.run.smithi186.stdout:/ 2024-03-28T15:01:47.702 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/fs/selinux 2024-03-28T15:01:47.702 INFO:teuthology.orchestra.run.smithi186.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T15:01:47.702 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/mqueue 2024-03-28T15:01:47.702 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/hugepages 2024-03-28T15:01:47.702 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/kernel/debug 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/kernel/tracing 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/fs/fuse/connections 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/sys/kernel/config 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/run/user/1000 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/ceph 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T15:01:47.703 INFO:teuthology.orchestra.run.smithi186.stdout:/run/netns 2024-03-28T15:01:47.704 INFO:teuthology.orchestra.run.smithi186.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T15:01:47.704 INFO:teuthology.orchestra.run.smithi186.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T15:01:47.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi186.stderr:2024-03-28T15:01:47.701+0000 7fb2b3840480 -1 init, newargv = 0x55dbb8898cb0 newargc=15 2024-03-28T15:01:47.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:47.705 DEBUG:teuthology.orchestra.run.smithi186:> ls /sys/fs/fuse/connections 2024-03-28T15:01:47.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi186.stderr:ceph-fuse[46497]: starting ceph client 2024-03-28T15:01:47.706 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi186.stderr:ceph-fuse[46497]: starting fuse 2024-03-28T15:01:47.720 INFO:teuthology.orchestra.run.smithi186.stdout:43 2024-03-28T15:01:47.721 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-28T15:01:47.721 DEBUG:teuthology.orchestra.run.smithi186:> sudo stdin-killer -- python3 -c ' 2024-03-28T15:01:47.721 DEBUG:teuthology.orchestra.run.smithi186:> import glob 2024-03-28T15:01:47.721 DEBUG:teuthology.orchestra.run.smithi186:> import re 2024-03-28T15:01:47.721 DEBUG:teuthology.orchestra.run.smithi186:> import os 2024-03-28T15:01:47.721 DEBUG:teuthology.orchestra.run.smithi186:> import subprocess 2024-03-28T15:01:47.721 DEBUG:teuthology.orchestra.run.smithi186:> 2024-03-28T15:01:47.721 DEBUG:teuthology.orchestra.run.smithi186:> def _find_admin_socket(client_name): 2024-03-28T15:01:47.721 DEBUG:teuthology.orchestra.run.smithi186:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-28T15:01:47.721 DEBUG:teuthology.orchestra.run.smithi186:> files = glob.glob(asok_path) 2024-03-28T15:01:47.721 DEBUG:teuthology.orchestra.run.smithi186:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-28T15:01:47.721 DEBUG:teuthology.orchestra.run.smithi186:> 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> # Given a non-glob path, it better be there 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> if "*" not in asok_path: 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> assert(len(files) == 1) 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> return files[0] 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> for f in files: 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> contents = proc_f.read() 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> if mountpoint in contents: 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> return f 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> 2024-03-28T15:01:47.722 DEBUG:teuthology.orchestra.run.smithi186:> print(_find_admin_socket("client.0")) 2024-03-28T15:01:47.723 DEBUG:teuthology.orchestra.run.smithi186:> ' 2024-03-28T15:01:47.843 INFO:teuthology.orchestra.run.smithi186.stdout:/var/run/ceph/ceph-client.0.46497.asok 2024-03-28T15:01:47.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T15:01:47.854 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46497.asok 2024-03-28T15:01:47.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:47.854 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46497.asok status 2024-03-28T15:01:47.990 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-28T15:01:47.990 INFO:teuthology.orchestra.run.smithi186.stdout: "metadata": { 2024-03-28T15:01:47.990 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_sha1": "e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15", 2024-03-28T15:01:47.990 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 19.0.0-2499-ge0e48ba6 (e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15) squid (dev)", 2024-03-28T15:01:47.990 INFO:teuthology.orchestra.run.smithi186.stdout: "entity_id": "0", 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "hostname": "smithi186", 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "pid": "46497", 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "root": "/" 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "dentry_count": 0, 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "dentry_pinned_count": 0, 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "id": 4698, 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "inst": { 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "name": { 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "client", 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "num": 4698 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-03-28T15:01:47.991 INFO:teuthology.orchestra.run.smithi186.stdout: "addr": { 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "v1", 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: "addr": "192.168.0.1:0", 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: "nonce": 604310927 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: "addr": { 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: "type": "v1", 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: "addr": "192.168.0.1:0", 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: "nonce": 604310927 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: "inst_str": "client.4698 192.168.0.1:0/604310927", 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: "addr_str": "192.168.0.1:0/604310927", 2024-03-28T15:01:47.992 INFO:teuthology.orchestra.run.smithi186.stdout: "inode_count": 1, 2024-03-28T15:01:47.993 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_epoch": 8, 2024-03-28T15:01:47.993 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_epoch": 20, 2024-03-28T15:01:47.993 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_epoch_barrier": 0, 2024-03-28T15:01:47.993 INFO:teuthology.orchestra.run.smithi186.stdout: "blocklisted": false, 2024-03-28T15:01:47.993 INFO:teuthology.orchestra.run.smithi186.stdout: "fs_name": "cephfs" 2024-03-28T15:01:47.993 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-28T15:01:47.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:47.999 DEBUG:teuthology.orchestra.run.smithi186:> stat --file-system '--printf=%T 2024-03-28T15:01:47.999 DEBUG:teuthology.orchestra.run.smithi186:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-28T15:01:48.014 INFO:teuthology.orchestra.run.smithi186.stdout:fuseblk 2024-03-28T15:01:48.014 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-28T15:01:48.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:01:48.014 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T15:01:48.084 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-03-28T15:01:48.096 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:48.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.227+0000 7f002dc2a640 1 Processor -- start 2024-03-28T15:01:48.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.227+0000 7f002dc2a640 1 -- start start 2024-03-28T15:01:48.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.227+0000 7f002dc2a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0028151990 0x7f00280c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.227+0000 7f002dc2a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00280b1e30 0x7f00280c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.227+0000 7f002dc2a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f002806caa0 con 0x7f00280c2d00 2024-03-28T15:01:48.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.227+0000 7f002dc2a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f00280c1da0 con 0x7f00280b1e30 2024-03-28T15:01:48.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.227+0000 7f002dc2a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f002806cd80 con 0x7f0028151990 2024-03-28T15:01:48.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.228+0000 7f0026ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00280b1e30 0x7f00280c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.228+0000 7f0026ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00280b1e30 0x7f00280c2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48726/0 (socket says 172.21.15.186:48726) 2024-03-28T15:01:48.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.228+0000 7f00277fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0028151990 0x7f00280c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.228+0000 7f0026ffd640 1 -- 172.21.15.186:0/2133796432 learned_addr learned my addr 172.21.15.186:0/2133796432 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:48.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.228+0000 7f0026ffd640 1 -- 172.21.15.186:0/2133796432 >> v2:172.21.15.191:3300/0 conn(0x7f0028151990 msgr2=0x7f00280c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:48.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.228+0000 7f0026ffd640 1 --2- 172.21.15.186:0/2133796432 >> v2:172.21.15.191:3300/0 conn(0x7f0028151990 0x7f00280c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.228+0000 7f0026ffd640 1 -- 172.21.15.186:0/2133796432 >> v1:172.21.15.191:6789/0 conn(0x7f00280c2d00 legacy=0x7f00280712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.228+0000 7f0026ffd640 1 -- 172.21.15.186:0/2133796432 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00280c1340 con 0x7f00280b1e30 2024-03-28T15:01:48.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f0026ffd640 1 --2- 172.21.15.186:0/2133796432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00280b1e30 0x7f00280c2770 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f002806fd90 tx=0x7f001c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc479d6e1246e25b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f00267fc640 1 -- 172.21.15.186:0/2133796432 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c02f850 con 0x7f00280b1e30 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f00267fc640 1 -- 172.21.15.186:0/2133796432 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f001c02f9f0 con 0x7f00280b1e30 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f00267fc640 1 -- 172.21.15.186:0/2133796432 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c02fdc0 con 0x7f00280b1e30 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f002dc2a640 1 -- 172.21.15.186:0/2133796432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00280b1e30 msgr2=0x7f00280c2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f002dc2a640 1 --2- 172.21.15.186:0/2133796432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00280b1e30 0x7f00280c2770 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f002806fd90 tx=0x7f001c02ba30 comp rx=0 tx=0).stop 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f002dc2a640 1 -- 172.21.15.186:0/2133796432 shutdown_connections 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f002dc2a640 1 --2- 172.21.15.186:0/2133796432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00280b1e30 0x7f00280c2770 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f002dc2a640 1 --2- 172.21.15.186:0/2133796432 >> v2:172.21.15.191:3300/0 conn(0x7f0028151990 0x7f00280c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f002dc2a640 1 -- 172.21.15.186:0/2133796432 >> 172.21.15.186:0/2133796432 conn(0x7f00280c8a20 msgr2=0x7f00280c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f002dc2a640 1 -- 172.21.15.186:0/2133796432 shutdown_connections 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.229+0000 7f002dc2a640 1 -- 172.21.15.186:0/2133796432 wait complete. 2024-03-28T15:01:48.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.230+0000 7f002dc2a640 1 Processor -- start 2024-03-28T15:01:48.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f002dc2a640 1 -- start start 2024-03-28T15:01:48.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f002dc2a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00280b1e30 0x7f00281ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f002dc2a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00280c2d00 0x7f002814ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f002dc2a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0028151990 0x7f00281558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f002dc2a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0028073dd0 con 0x7f00280c2d00 2024-03-28T15:01:48.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f002dc2a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f00280c1900 con 0x7f00280b1e30 2024-03-28T15:01:48.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f002dc2a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f0028073f50 con 0x7f0028151990 2024-03-28T15:01:48.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f0026ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00280c2d00 0x7f002814ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f0026ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00280c2d00 0x7f002814ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48736/0 (socket says 172.21.15.186:48736) 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f0026ffd640 1 -- 172.21.15.186:0/453224543 learned_addr learned my addr 172.21.15.186:0/453224543 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f00277fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00280b1e30 0x7f00281ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.231+0000 7f0027fff640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0028151990 0x7f00281558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f0026ffd640 1 -- 172.21.15.186:0/453224543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0028151990 msgr2=0x7f00281558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f0026ffd640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0028151990 0x7f00281558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f0026ffd640 1 -- 172.21.15.186:0/453224543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00280b1e30 msgr2=0x7f00281ee5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f0026ffd640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00280b1e30 0x7f00281ee5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f0026ffd640 1 -- 172.21.15.186:0/453224543 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0028156c00 con 0x7f00280c2d00 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f00277fe640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00280b1e30 0x7f00281ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f0026ffd640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00280c2d00 0x7f002814ef50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f001c000c00 tx=0x7f001c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f000ffff640 1 -- 172.21.15.186:0/453224543 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c047070 con 0x7f00280c2d00 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f000ffff640 1 -- 172.21.15.186:0/453224543 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f001c037d30 con 0x7f00280c2d00 2024-03-28T15:01:48.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f002dc2a640 1 -- 172.21.15.186:0/453224543 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0028156ee0 con 0x7f00280c2d00 2024-03-28T15:01:48.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f000ffff640 1 -- 172.21.15.186:0/453224543 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c0426c0 con 0x7f00280c2d00 2024-03-28T15:01:48.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.232+0000 7f002dc2a640 1 -- 172.21.15.186:0/453224543 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00281574a0 con 0x7f00280c2d00 2024-03-28T15:01:48.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.233+0000 7f002dc2a640 1 -- 172.21.15.186:0/453224543 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0028069500 con 0x7f00280c2d00 2024-03-28T15:01:48.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.233+0000 7f000ffff640 1 -- 172.21.15.186:0/453224543 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f001c04a070 con 0x7f00280c2d00 2024-03-28T15:01:48.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.234+0000 7f000ffff640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f001406c530 0x7f001406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.234+0000 7f000ffff640 1 -- 172.21.15.186:0/453224543 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f001c0ad830 con 0x7f00280c2d00 2024-03-28T15:01:48.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.234+0000 7f00277fe640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f001406c530 0x7f001406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.234+0000 7f00277fe640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f001406c530 0x7f001406e9f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0018009f30 tx=0x7f0018009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:48.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.238+0000 7f000ffff640 1 -- 172.21.15.186:0/453224543 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f001c033030 con 0x7f00280c2d00 2024-03-28T15:01:48.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.416+0000 7f002dc2a640 1 -- 172.21.15.186:0/453224543 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00280737d0 con 0x7f00280c2d00 2024-03-28T15:01:48.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.417+0000 7f000ffff640 1 -- 172.21.15.186:0/453224543 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f001c07b890 con 0x7f00280c2d00 2024-03-28T15:01:48.417 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:48.417 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:33.099254+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:48.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 -- 172.21.15.186:0/453224543 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f001406c530 msgr2=0x7f001406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f001406c530 0x7f001406e9f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0018009f30 tx=0x7f0018009190 comp rx=0 tx=0).stop 2024-03-28T15:01:48.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 -- 172.21.15.186:0/453224543 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00280c2d00 msgr2=0x7f002814ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00280c2d00 0x7f002814ef50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f001c000c00 tx=0x7f001c004730 comp rx=0 tx=0).stop 2024-03-28T15:01:48.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 -- 172.21.15.186:0/453224543 shutdown_connections 2024-03-28T15:01:48.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f001406c530 0x7f001406e9f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0028151990 0x7f00281558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00280c2d00 0x7f002814ef50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 --2- 172.21.15.186:0/453224543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00280b1e30 0x7f00281ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 -- 172.21.15.186:0/453224543 >> 172.21.15.186:0/453224543 conn(0x7f00280c8a20 msgr2=0x7f0028150240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:48.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 -- 172.21.15.186:0/453224543 shutdown_connections 2024-03-28T15:01:48.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.419+0000 7f002dc2a640 1 -- 172.21.15.186:0/453224543 wait complete. 2024-03-28T15:01:48.430 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:01:18.661274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:01:29.512445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:01:30.388340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:01:48.431 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:01:48.534 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.534+0000 7fec8c04e640 1 Processor -- start 2024-03-28T15:01:48.534 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.534+0000 7fec8c04e640 1 -- start start 2024-03-28T15:01:48.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec8c04e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fec84157fb0 0x7fec8406b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec8c04e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 0x7fec8415df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec8c04e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec8406f710 con 0x7fec840c89b0 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec8c04e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fec8415ee00 con 0x7fec84157fb0 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec8c04e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fec840b3560 con 0x7fec84070dd0 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec8a5eb640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fec840c89b0 0x7fec8415a9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51466/0 (socket says 172.21.15.186:51466) 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec8a5eb640 1 -- 172.21.15.186:0/3482617932 learned_addr learned my addr 172.21.15.186:0/3482617932 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec89dea640 1 --2- 172.21.15.186:0/3482617932 >> v2:172.21.15.191:3300/0 conn(0x7fec84157fb0 0x7fec8406b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec895e9640 1 --2- 172.21.15.186:0/3482617932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 0x7fec8415df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec89dea640 1 -- 172.21.15.186:0/3482617932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 msgr2=0x7fec8415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec89dea640 1 --2- 172.21.15.186:0/3482617932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 0x7fec8415df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec88de8640 1 -- 172.21.15.186:0/3482617932 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699416982 0 0) 0x7fec8406f710 con 0x7fec840c89b0 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec89dea640 1 -- 172.21.15.186:0/3482617932 >> v1:172.21.15.186:6789/0 conn(0x7fec840c89b0 legacy=0x7fec8415a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec89dea640 1 -- 172.21.15.186:0/3482617932 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8415eb20 con 0x7fec84157fb0 2024-03-28T15:01:48.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.535+0000 7fec895e9640 1 --2- 172.21.15.186:0/3482617932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 0x7fec8415df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.536+0000 7fec89dea640 1 --2- 172.21.15.186:0/3482617932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec84157fb0 0x7fec8406b560 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fec800097a0 tx=0x7fec8002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bf3d67a1b3e764d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.536+0000 7fec88de8640 1 -- 172.21.15.186:0/3482617932 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80035020 con 0x7fec84157fb0 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.536+0000 7fec88de8640 1 -- 172.21.15.186:0/3482617932 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec80031830 con 0x7fec84157fb0 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.536+0000 7fec88de8640 1 -- 172.21.15.186:0/3482617932 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80031c00 con 0x7fec84157fb0 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.537+0000 7fec8c04e640 1 -- 172.21.15.186:0/3482617932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec84157fb0 msgr2=0x7fec8406b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.537+0000 7fec8c04e640 1 --2- 172.21.15.186:0/3482617932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec84157fb0 0x7fec8406b560 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fec800097a0 tx=0x7fec8002b8d0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.537+0000 7fec8c04e640 1 -- 172.21.15.186:0/3482617932 shutdown_connections 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.537+0000 7fec8c04e640 1 --2- 172.21.15.186:0/3482617932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 0x7fec8415df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.537+0000 7fec8c04e640 1 --2- 172.21.15.186:0/3482617932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec84157fb0 0x7fec8406b560 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.537+0000 7fec8c04e640 1 -- 172.21.15.186:0/3482617932 >> 172.21.15.186:0/3482617932 conn(0x7fec84073610 msgr2=0x7fec8406d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.537+0000 7fec8c04e640 1 -- 172.21.15.186:0/3482617932 shutdown_connections 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.537+0000 7fec8c04e640 1 -- 172.21.15.186:0/3482617932 wait complete. 2024-03-28T15:01:48.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.538+0000 7fec8c04e640 1 Processor -- start 2024-03-28T15:01:48.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.538+0000 7fec8c04e640 1 -- start start 2024-03-28T15:01:48.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.538+0000 7fec8c04e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 0x7fec8414c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec8c04e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec840c89b0 0x7fec841f1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec8c04e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec84157fb0 0x7fec840ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec8c04e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec84070820 con 0x7fec840c89b0 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec8c04e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fec8406f9b0 con 0x7fec84157fb0 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec8c04e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fec84158390 con 0x7fec84070dd0 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec895e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec840c89b0 0x7fec841f1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec895e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec840c89b0 0x7fec841f1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48748/0 (socket says 172.21.15.186:48748) 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec89dea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 0x7fec8414c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec895e9640 1 -- 172.21.15.186:0/648160105 learned_addr learned my addr 172.21.15.186:0/648160105 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec8a5eb640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec84157fb0 0x7fec840ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec89dea640 1 -- 172.21.15.186:0/648160105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec84157fb0 msgr2=0x7fec840ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec89dea640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec84157fb0 0x7fec840ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec89dea640 1 -- 172.21.15.186:0/648160105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec840c89b0 msgr2=0x7fec841f1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec89dea640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec840c89b0 0x7fec841f1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec89dea640 1 -- 172.21.15.186:0/648160105 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec84156950 con 0x7fec84070dd0 2024-03-28T15:01:48.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec89dea640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 0x7fec8414c9b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fec80033000 tx=0x7fec800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:48.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec7a7fc640 1 -- 172.21.15.186:0/648160105 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80035070 con 0x7fec84070dd0 2024-03-28T15:01:48.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec7a7fc640 1 -- 172.21.15.186:0/648160105 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec80004230 con 0x7fec84070dd0 2024-03-28T15:01:48.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec8c04e640 1 -- 172.21.15.186:0/648160105 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8414e700 con 0x7fec84070dd0 2024-03-28T15:01:48.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec7a7fc640 1 -- 172.21.15.186:0/648160105 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8003a6f0 con 0x7fec84070dd0 2024-03-28T15:01:48.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.539+0000 7fec8c04e640 1 -- 172.21.15.186:0/648160105 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fec84156bc0 con 0x7fec84070dd0 2024-03-28T15:01:48.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.540+0000 7fec8c04e640 1 -- 172.21.15.186:0/648160105 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec840681e0 con 0x7fec84070dd0 2024-03-28T15:01:48.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.541+0000 7fec7a7fc640 1 -- 172.21.15.186:0/648160105 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fec80038020 con 0x7fec84070dd0 2024-03-28T15:01:48.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.541+0000 7fec7a7fc640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fec6806c560 0x7fec6806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.541+0000 7fec895e9640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fec6806c560 0x7fec6806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.541+0000 7fec7a7fc640 1 -- 172.21.15.186:0/648160105 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fec800ad230 con 0x7fec84070dd0 2024-03-28T15:01:48.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.541+0000 7fec895e9640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fec6806c560 0x7fec6806ea20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fec740095e0 tx=0x7fec74009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:48.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.546+0000 7fec7a7fc640 1 -- 172.21.15.186:0/648160105 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fec80040070 con 0x7fec84070dd0 2024-03-28T15:01:48.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.722+0000 7fec8c04e640 1 -- 172.21.15.186:0/648160105 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fec84069570 con 0x7fec84070dd0 2024-03-28T15:01:48.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.723+0000 7fec7a7fc640 1 -- 172.21.15.186:0/648160105 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fec8007b0f0 con 0x7fec84070dd0 2024-03-28T15:01:48.723 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 1 2024-03-28T15:01:48.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 -- 172.21.15.186:0/648160105 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fec6806c560 msgr2=0x7fec6806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fec6806c560 0x7fec6806ea20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fec740095e0 tx=0x7fec74009190 comp rx=0 tx=0).stop 2024-03-28T15:01:48.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 -- 172.21.15.186:0/648160105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 msgr2=0x7fec8414c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 0x7fec8414c9b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fec80033000 tx=0x7fec800046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 -- 172.21.15.186:0/648160105 shutdown_connections 2024-03-28T15:01:48.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fec6806c560 0x7fec6806ea20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec84157fb0 0x7fec840ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec840c89b0 0x7fec841f1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 --2- 172.21.15.186:0/648160105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec84070dd0 0x7fec8414c9b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 -- 172.21.15.186:0/648160105 >> 172.21.15.186:0/648160105 conn(0x7fec84073610 msgr2=0x7fec84154a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:48.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 -- 172.21.15.186:0/648160105 shutdown_connections 2024-03-28T15:01:48.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.725+0000 7fec8c04e640 1 -- 172.21.15.186:0/648160105 wait complete. 2024-03-28T15:01:48.736 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:01:48.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.840+0000 7f8585a4e640 1 Processor -- start 2024-03-28T15:01:48.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.840+0000 7f8585a4e640 1 -- start start 2024-03-28T15:01:48.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.840+0000 7f8585a4e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8580154a10 0x7f85800cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.840+0000 7f8585a4e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f85800cce90 0x7f85800cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.840+0000 7f8585a4e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f858015de90 con 0x7f858015b2c0 2024-03-28T15:01:48.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.840+0000 7f8585a4e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f85800c5ae0 con 0x7f8580154a10 2024-03-28T15:01:48.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.840+0000 7f8585a4e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f858015e750 con 0x7f85800cce90 2024-03-28T15:01:48.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.841+0000 7f857effd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8580154a10 0x7f85800cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.841+0000 7f857effd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8580154a10 0x7f85800cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42254/0 (socket says 172.21.15.186:42254) 2024-03-28T15:01:48.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.841+0000 7f857f7fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f85800cce90 0x7f85800cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.841+0000 7f857effd640 1 -- 172.21.15.186:0/2755886847 learned_addr learned my addr 172.21.15.186:0/2755886847 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:48.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.841+0000 7f857f7fe640 1 -- 172.21.15.186:0/2755886847 >> v2:172.21.15.186:3300/0 conn(0x7f8580154a10 msgr2=0x7f85800cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:48.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.841+0000 7f857f7fe640 1 --2- 172.21.15.186:0/2755886847 >> v2:172.21.15.186:3300/0 conn(0x7f8580154a10 0x7f85800cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.841+0000 7f857f7fe640 1 -- 172.21.15.186:0/2755886847 >> v1:172.21.15.191:6789/0 conn(0x7f858015b2c0 legacy=0x7f85800c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.841+0000 7f857f7fe640 1 -- 172.21.15.186:0/2755886847 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85800cb6d0 con 0x7f85800cce90 2024-03-28T15:01:48.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.841+0000 7f857dffb640 1 -- 172.21.15.186:0/2755886847 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2500757786 0 0) 0x7f858015de90 con 0x7f858015b2c0 2024-03-28T15:01:48.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f857f7fe640 1 --2- 172.21.15.186:0/2755886847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85800cce90 0x7f85800cd2c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f858008e260 tx=0x7f857802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6becaa46cd8d02d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:48.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f857dffb640 1 -- 172.21.15.186:0/2755886847 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8578039070 con 0x7f85800cce90 2024-03-28T15:01:48.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f857dffb640 1 -- 172.21.15.186:0/2755886847 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f857802cd00 con 0x7f85800cce90 2024-03-28T15:01:48.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f857dffb640 1 -- 172.21.15.186:0/2755886847 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8578034660 con 0x7f85800cce90 2024-03-28T15:01:48.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f8585a4e640 1 -- 172.21.15.186:0/2755886847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85800cce90 msgr2=0x7f85800cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f8585a4e640 1 --2- 172.21.15.186:0/2755886847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85800cce90 0x7f85800cd2c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f858008e260 tx=0x7f857802bf50 comp rx=0 tx=0).stop 2024-03-28T15:01:48.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f8585a4e640 1 -- 172.21.15.186:0/2755886847 shutdown_connections 2024-03-28T15:01:48.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f8585a4e640 1 --2- 172.21.15.186:0/2755886847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85800cce90 0x7f85800cd2c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f8585a4e640 1 --2- 172.21.15.186:0/2755886847 >> v2:172.21.15.186:3300/0 conn(0x7f8580154a10 0x7f85800cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f8585a4e640 1 -- 172.21.15.186:0/2755886847 >> 172.21.15.186:0/2755886847 conn(0x7f858006b640 msgr2=0x7f858006ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:48.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f8585a4e640 1 -- 172.21.15.186:0/2755886847 shutdown_connections 2024-03-28T15:01:48.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.842+0000 7f8585a4e640 1 -- 172.21.15.186:0/2755886847 wait complete. 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f8585a4e640 1 Processor -- start 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f8585a4e640 1 -- start start 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f8585a4e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85800cce90 0x7f85800cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f8585a4e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8580154a10 0x7f858015b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f8585a4e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f858015b2c0 0x7f858015bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f8585a4e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f85800cb9b0 con 0x7f8580154a10 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f8585a4e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f858015e3e0 con 0x7f85800cce90 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f8585a4e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f858015b710 con 0x7f858015b2c0 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f857e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8580154a10 0x7f858015b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f857e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8580154a10 0x7f858015b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42266/0 (socket says 172.21.15.186:42266) 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f857e7fc640 1 -- 172.21.15.186:0/3492687612 learned_addr learned my addr 172.21.15.186:0/3492687612 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.844+0000 7f857effd640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85800cce90 0x7f85800cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f857f7fe640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f858015b2c0 0x7f858015bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f857e7fc640 1 -- 172.21.15.186:0/3492687612 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f858015b2c0 msgr2=0x7f858015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f857e7fc640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f858015b2c0 0x7f858015bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f857e7fc640 1 -- 172.21.15.186:0/3492687612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85800cce90 msgr2=0x7f85800cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:48.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f857e7fc640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85800cce90 0x7f85800cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:48.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f857e7fc640 1 -- 172.21.15.186:0/3492687612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85800c9dd0 con 0x7f8580154a10 2024-03-28T15:01:48.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f857effd640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85800cce90 0x7f85800cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:48.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f857e7fc640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8580154a10 0x7f858015b9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f857000c4e0 tx=0x7f857000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:48.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f8567fff640 1 -- 172.21.15.186:0/3492687612 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570017070 con 0x7f8580154a10 2024-03-28T15:01:48.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f8567fff640 1 -- 172.21.15.186:0/3492687612 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85700035f0 con 0x7f8580154a10 2024-03-28T15:01:48.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f8585a4e640 1 -- 172.21.15.186:0/3492687612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85800ca0b0 con 0x7f8580154a10 2024-03-28T15:01:48.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f8567fff640 1 -- 172.21.15.186:0/3492687612 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85700039e0 con 0x7f8580154a10 2024-03-28T15:01:48.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.845+0000 7f8585a4e640 1 -- 172.21.15.186:0/3492687612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85800d06a0 con 0x7f8580154a10 2024-03-28T15:01:48.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.846+0000 7f8585a4e640 1 -- 172.21.15.186:0/3492687612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8580069570 con 0x7f8580154a10 2024-03-28T15:01:48.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.847+0000 7f8567fff640 1 -- 172.21.15.186:0/3492687612 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f857001a070 con 0x7f8580154a10 2024-03-28T15:01:48.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.847+0000 7f8567fff640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f856c06c530 0x7f856c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:48.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.847+0000 7f8567fff640 1 -- 172.21.15.186:0/3492687612 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f857008a990 con 0x7f8580154a10 2024-03-28T15:01:48.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.847+0000 7f857effd640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f856c06c530 0x7f856c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:48.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.847+0000 7f857effd640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f856c06c530 0x7f856c06e9f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8574009fc0 tx=0x7f8574009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:48.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:48.851+0000 7f8567fff640 1 -- 172.21.15.186:0/3492687612 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8570058960 con 0x7f8580154a10 2024-03-28T15:01:49.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.026+0000 7f8585a4e640 1 -- 172.21.15.186:0/3492687612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8580159830 con 0x7f8580154a10 2024-03-28T15:01:49.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.027+0000 7f8567fff640 1 -- 172.21.15.186:0/3492687612 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f857005cc60 con 0x7f8580154a10 2024-03-28T15:01:49.027 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 32 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.028+0000 7f8585a4e640 1 -- 172.21.15.186:0/3492687612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f856c06c530 msgr2=0x7f856c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.028+0000 7f8585a4e640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f856c06c530 0x7f856c06e9f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8574009fc0 tx=0x7f8574009240 comp rx=0 tx=0).stop 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.028+0000 7f8585a4e640 1 -- 172.21.15.186:0/3492687612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8580154a10 msgr2=0x7f858015b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.028+0000 7f8585a4e640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8580154a10 0x7f858015b9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f857000c4e0 tx=0x7f857000c9a0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.028+0000 7f8585a4e640 1 -- 172.21.15.186:0/3492687612 shutdown_connections 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.028+0000 7f8585a4e640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f856c06c530 0x7f856c06e9f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.028+0000 7f8585a4e640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f858015b2c0 0x7f858015bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.028+0000 7f8585a4e640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8580154a10 0x7f858015b9c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.028+0000 7f8585a4e640 1 --2- 172.21.15.186:0/3492687612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85800cce90 0x7f85800cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.028+0000 7f8585a4e640 1 -- 172.21.15.186:0/3492687612 >> 172.21.15.186:0/3492687612 conn(0x7f858006b640 msgr2=0x7f8580153be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.029+0000 7f8585a4e640 1 -- 172.21.15.186:0/3492687612 shutdown_connections 2024-03-28T15:01:49.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.029+0000 7f8585a4e640 1 -- 172.21.15.186:0/3492687612 wait complete. 2024-03-28T15:01:49.040 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:01:49.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.144+0000 7faab1db5640 1 Processor -- start 2024-03-28T15:01:49.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.144+0000 7faab1db5640 1 -- start start 2024-03-28T15:01:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.144+0000 7faab1db5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faaac154b60 0x7faaac15a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.144+0000 7faab1db5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cb3f0 0x7faaac071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.144+0000 7faab1db5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaac0cd620 con 0x7faaac0cafd0 2024-03-28T15:01:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.144+0000 7faab1db5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7faaac072820 con 0x7faaac154b60 2024-03-28T15:01:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.144+0000 7faab1db5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7faaac15b760 con 0x7faaac0cb3f0 2024-03-28T15:01:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.145+0000 7faaab7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faaac154b60 0x7faaac15a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.145+0000 7faaab7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faaac154b60 0x7faaac15a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42268/0 (socket says 172.21.15.186:42268) 2024-03-28T15:01:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.145+0000 7faaaaffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7faaac0cafd0 0x7faaac0ccf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49718/0 (socket says 172.21.15.186:49718) 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.145+0000 7faaab7fe640 1 -- 172.21.15.186:0/214909267 learned_addr learned my addr 172.21.15.186:0/214909267 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.145+0000 7faaabfff640 1 --2- 172.21.15.186:0/214909267 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cb3f0 0x7faaac071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.145+0000 7faaabfff640 1 -- 172.21.15.186:0/214909267 >> v2:172.21.15.186:3300/0 conn(0x7faaac154b60 msgr2=0x7faaac15a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.145+0000 7faaabfff640 1 --2- 172.21.15.186:0/214909267 >> v2:172.21.15.186:3300/0 conn(0x7faaac154b60 0x7faaac15a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.145+0000 7faaabfff640 1 -- 172.21.15.186:0/214909267 >> v1:172.21.15.186:6789/0 conn(0x7faaac0cafd0 legacy=0x7faaac0ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.145+0000 7faaabfff640 1 -- 172.21.15.186:0/214909267 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaac15aec0 con 0x7faaac0cb3f0 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faaabfff640 1 --2- 172.21.15.186:0/214909267 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cb3f0 0x7faaac071a80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faaa4009d90 tx=0x7faaa402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b19e001d31d8d18 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faaaa7fc640 1 -- 172.21.15.186:0/214909267 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa4039070 con 0x7faaac0cb3f0 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faaaa7fc640 1 -- 172.21.15.186:0/214909267 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa402cce0 con 0x7faaac0cb3f0 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faaaa7fc640 1 -- 172.21.15.186:0/214909267 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa4034660 con 0x7faaac0cb3f0 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faab1db5640 1 -- 172.21.15.186:0/214909267 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cb3f0 msgr2=0x7faaac071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faab1db5640 1 --2- 172.21.15.186:0/214909267 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cb3f0 0x7faaac071a80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faaa4009d90 tx=0x7faaa402bf30 comp rx=0 tx=0).stop 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faab1db5640 1 -- 172.21.15.186:0/214909267 shutdown_connections 2024-03-28T15:01:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faab1db5640 1 --2- 172.21.15.186:0/214909267 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cb3f0 0x7faaac071a80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faab1db5640 1 --2- 172.21.15.186:0/214909267 >> v2:172.21.15.186:3300/0 conn(0x7faaac154b60 0x7faaac15a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faab1db5640 1 -- 172.21.15.186:0/214909267 >> 172.21.15.186:0/214909267 conn(0x7faaac073b80 msgr2=0x7faaac06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faab1db5640 1 -- 172.21.15.186:0/214909267 shutdown_connections 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.146+0000 7faab1db5640 1 -- 172.21.15.186:0/214909267 wait complete. 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.147+0000 7faab1db5640 1 Processor -- start 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faab1db5640 1 -- start start 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faab1db5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cafd0 0x7faaac0c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faab1db5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaac0cb3f0 0x7faaac1525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faab1db5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac154b60 0x7faaac1f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faab1db5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faaac0cc710 con 0x7faaac154b60 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faab1db5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faaac15b480 con 0x7faaac0cb3f0 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faab1db5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7faaac0c53b0 con 0x7faaac0cafd0 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faaab7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cafd0 0x7faaac0c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faaabfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac154b60 0x7faaac1f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faaab7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cafd0 0x7faaac0c89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:46724/0 (socket says 172.21.15.186:46724) 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faaab7fe640 1 -- 172.21.15.186:0/2531084573 learned_addr learned my addr 172.21.15.186:0/2531084573 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faaabfff640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac154b60 0x7faaac1f06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42284/0 (socket says 172.21.15.186:42284) 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.148+0000 7faaaaffd640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaac0cb3f0 0x7faaac1525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faaab7fe640 1 -- 172.21.15.186:0/2531084573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaac0cb3f0 msgr2=0x7faaac1525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faaab7fe640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaac0cb3f0 0x7faaac1525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faaab7fe640 1 -- 172.21.15.186:0/2531084573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac154b60 msgr2=0x7faaac1f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faaab7fe640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac154b60 0x7faaac1f06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faaab7fe640 1 -- 172.21.15.186:0/2531084573 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaac1f11e0 con 0x7faaac0cafd0 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faaabfff640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac154b60 0x7faaac1f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faaab7fe640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cafd0 0x7faaac0c89b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faa9c00e870 tx=0x7faa9c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faa93fff640 1 -- 172.21.15.186:0/2531084573 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c017070 con 0x7faaac0cafd0 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faa93fff640 1 -- 172.21.15.186:0/2531084573 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa9c00c8e0 con 0x7faaac0cafd0 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faab1db5640 1 -- 172.21.15.186:0/2531084573 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaac1f14c0 con 0x7faaac0cafd0 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faa93fff640 1 -- 172.21.15.186:0/2531084573 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c00ccd0 con 0x7faaac0cafd0 2024-03-28T15:01:49.153 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.149+0000 7faab1db5640 1 -- 172.21.15.186:0/2531084573 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faaac149540 con 0x7faaac0cafd0 2024-03-28T15:01:49.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.150+0000 7faab1db5640 1 -- 172.21.15.186:0/2531084573 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaac0cc9f0 con 0x7faaac0cafd0 2024-03-28T15:01:49.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.151+0000 7faa93fff640 1 -- 172.21.15.186:0/2531084573 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7faa9c01a070 con 0x7faaac0cafd0 2024-03-28T15:01:49.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.151+0000 7faa93fff640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa9806c530 0x7faa9806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.151+0000 7faa93fff640 1 -- 172.21.15.186:0/2531084573 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7faa9c08b950 con 0x7faaac0cafd0 2024-03-28T15:01:49.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.151+0000 7faaaaffd640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa9806c530 0x7faa9806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.151+0000 7faaaaffd640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa9806c530 0x7faa9806e9f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faaac0b61c0 tx=0x7faaa0007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:49.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.160+0000 7faa93fff640 1 -- 172.21.15.186:0/2531084573 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7faaac0cc9f0 con 0x7faaac0cafd0 2024-03-28T15:01:49.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.334+0000 7faab1db5640 1 -- 172.21.15.186:0/2531084573 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faaac0686d0 con 0x7faaac0cafd0 2024-03-28T15:01:49.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.334+0000 7faa93fff640 1 -- 172.21.15.186:0/2531084573 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7faa9c059930 con 0x7faaac0cafd0 2024-03-28T15:01:49.335 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 64 2024-03-28T15:01:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 -- 172.21.15.186:0/2531084573 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa9806c530 msgr2=0x7faa9806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa9806c530 0x7faa9806e9f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faaac0b61c0 tx=0x7faaa0007e80 comp rx=0 tx=0).stop 2024-03-28T15:01:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 -- 172.21.15.186:0/2531084573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cafd0 msgr2=0x7faaac0c89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cafd0 0x7faaac0c89b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faa9c00e870 tx=0x7faa9c00ed30 comp rx=0 tx=0).stop 2024-03-28T15:01:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 -- 172.21.15.186:0/2531084573 shutdown_connections 2024-03-28T15:01:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa9806c530 0x7faa9806e9f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac154b60 0x7faaac1f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaac0cb3f0 0x7faaac1525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 --2- 172.21.15.186:0/2531084573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0cafd0 0x7faaac0c89b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 -- 172.21.15.186:0/2531084573 >> 172.21.15.186:0/2531084573 conn(0x7faaac073b80 msgr2=0x7faaac0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 -- 172.21.15.186:0/2531084573 shutdown_connections 2024-03-28T15:01:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.336+0000 7faab1db5640 1 -- 172.21.15.186:0/2531084573 wait complete. 2024-03-28T15:01:49.347 INFO:tasks.mds_thrash:mds thrasher using random seed: 1711638109 2024-03-28T15:01:49.347 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:49.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.449+0000 7f5748379640 1 Processor -- start 2024-03-28T15:01:49.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.449+0000 7f5748379640 1 -- start start 2024-03-28T15:01:49.450 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5748379640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5740150ed0 0x7f57400c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.450 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5748379640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f57400733f0 0x7f57400b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.450 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5748379640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57400c1b40 con 0x7f57401512b0 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5748379640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f57400c1e20 con 0x7f57400733f0 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5748379640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f57400c18a0 con 0x7f5740150ed0 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5746916640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f57400733f0 0x7f57400b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5746916640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f57400733f0 0x7f57400b1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42288/0 (socket says 172.21.15.186:42288) 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5745914640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f57401512b0 0x7f5740156d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49732/0 (socket says 172.21.15.186:49732) 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5746916640 1 -- 172.21.15.186:0/3640829255 learned_addr learned my addr 172.21.15.186:0/3640829255 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5746115640 1 --2- 172.21.15.186:0/3640829255 >> v2:172.21.15.191:3300/0 conn(0x7f5740150ed0 0x7f57400c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5746916640 1 -- 172.21.15.186:0/3640829255 >> v2:172.21.15.191:3300/0 conn(0x7f5740150ed0 msgr2=0x7f57400c0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5746916640 1 --2- 172.21.15.186:0/3640829255 >> v2:172.21.15.191:3300/0 conn(0x7f5740150ed0 0x7f57400c0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5746916640 1 -- 172.21.15.186:0/3640829255 >> v1:172.21.15.186:6789/0 conn(0x7f57401512b0 legacy=0x7f5740156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5745113640 1 -- 172.21.15.186:0/3640829255 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2496745821 0 0) 0x7f57400c1b40 con 0x7f57401512b0 2024-03-28T15:01:49.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.450+0000 7f5746916640 1 -- 172.21.15.186:0/3640829255 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57400c15c0 con 0x7f57400733f0 2024-03-28T15:01:49.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.451+0000 7f5746916640 1 --2- 172.21.15.186:0/3640829255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57400733f0 0x7f57400b1750 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5740067be0 tx=0x7f573802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7e71e024d65185 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:49.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.451+0000 7f5745113640 1 -- 172.21.15.186:0/3640829255 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573802cbb0 con 0x7f57400733f0 2024-03-28T15:01:49.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.451+0000 7f5745113640 1 -- 172.21.15.186:0/3640829255 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f573802cd50 con 0x7f57400733f0 2024-03-28T15:01:49.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.451+0000 7f5745113640 1 -- 172.21.15.186:0/3640829255 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738034610 con 0x7f57400733f0 2024-03-28T15:01:49.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.451+0000 7f5748379640 1 -- 172.21.15.186:0/3640829255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57400733f0 msgr2=0x7f57400b1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.451+0000 7f5748379640 1 --2- 172.21.15.186:0/3640829255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57400733f0 0x7f57400b1750 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5740067be0 tx=0x7f573802c010 comp rx=0 tx=0).stop 2024-03-28T15:01:49.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.452+0000 7f5748379640 1 -- 172.21.15.186:0/3640829255 shutdown_connections 2024-03-28T15:01:49.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.452+0000 7f5748379640 1 --2- 172.21.15.186:0/3640829255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57400733f0 0x7f57400b1750 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.452+0000 7f5748379640 1 --2- 172.21.15.186:0/3640829255 >> v2:172.21.15.191:3300/0 conn(0x7f5740150ed0 0x7f57400c0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.452+0000 7f5748379640 1 -- 172.21.15.186:0/3640829255 >> 172.21.15.186:0/3640829255 conn(0x7f57400c8a20 msgr2=0x7f57400c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:49.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.452+0000 7f5748379640 1 -- 172.21.15.186:0/3640829255 shutdown_connections 2024-03-28T15:01:49.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.452+0000 7f5748379640 1 -- 172.21.15.186:0/3640829255 wait complete. 2024-03-28T15:01:49.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.453+0000 7f5748379640 1 Processor -- start 2024-03-28T15:01:49.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.453+0000 7f5748379640 1 -- start start 2024-03-28T15:01:49.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.453+0000 7f5748379640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57400733f0 0x7f57401ecdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.453+0000 7f5748379640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5740150ed0 0x7f57401ed7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.453+0000 7f5748379640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57401512b0 0x7f574014aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.453+0000 7f5748379640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f574006e5b0 con 0x7f57401512b0 2024-03-28T15:01:49.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.453+0000 7f5748379640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5740072f60 con 0x7f5740150ed0 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.453+0000 7f5748379640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f57400712b0 con 0x7f57400733f0 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5746916640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57401512b0 0x7f574014aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5746916640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57401512b0 0x7f574014aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42304/0 (socket says 172.21.15.186:42304) 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5746916640 1 -- 172.21.15.186:0/4181790533 learned_addr learned my addr 172.21.15.186:0/4181790533 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5746115640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57400733f0 0x7f57401ecdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5745914640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5740150ed0 0x7f57401ed7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5746916640 1 -- 172.21.15.186:0/4181790533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57400733f0 msgr2=0x7f57401ecdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5746916640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57400733f0 0x7f57401ecdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5746916640 1 -- 172.21.15.186:0/4181790533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5740150ed0 msgr2=0x7f57401ed7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5746916640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5740150ed0 0x7f57401ed7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5746916640 1 -- 172.21.15.186:0/4181790533 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574014fea0 con 0x7f57401512b0 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5746916640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57401512b0 0x7f574014aa00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f573802c620 tx=0x7f5738004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5736ffd640 1 -- 172.21.15.186:0/4181790533 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738043020 con 0x7f57401512b0 2024-03-28T15:01:49.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5736ffd640 1 -- 172.21.15.186:0/4181790533 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f573803e450 con 0x7f57401512b0 2024-03-28T15:01:49.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5748379640 1 -- 172.21.15.186:0/4181790533 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57400c35c0 con 0x7f57401512b0 2024-03-28T15:01:49.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5736ffd640 1 -- 172.21.15.186:0/4181790533 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738009510 con 0x7f57401512b0 2024-03-28T15:01:49.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.454+0000 7f5748379640 1 -- 172.21.15.186:0/4181790533 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57400c3b80 con 0x7f57401512b0 2024-03-28T15:01:49.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.455+0000 7f5748379640 1 -- 172.21.15.186:0/4181790533 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5740069500 con 0x7f57401512b0 2024-03-28T15:01:49.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.456+0000 7f5736ffd640 1 -- 172.21.15.186:0/4181790533 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5738046020 con 0x7f57401512b0 2024-03-28T15:01:49.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.456+0000 7f5736ffd640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f572406c530 0x7f572406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.456+0000 7f5736ffd640 1 -- 172.21.15.186:0/4181790533 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f57380adda0 con 0x7f57401512b0 2024-03-28T15:01:49.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.456+0000 7f5746115640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f572406c530 0x7f572406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.456+0000 7f5746115640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f572406c530 0x7f572406e9f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f573c009b40 tx=0x7f573c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:49.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.460+0000 7f5736ffd640 1 -- 172.21.15.186:0/4181790533 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f573804d030 con 0x7f57401512b0 2024-03-28T15:01:49.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.636+0000 7f5748379640 1 -- 172.21.15.186:0/4181790533 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57401edd10 con 0x7f57401512b0 2024-03-28T15:01:49.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.637+0000 7f5736ffd640 1 -- 172.21.15.186:0/4181790533 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f573807bce0 con 0x7f57401512b0 2024-03-28T15:01:49.637 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:49.637 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:33.099254+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:49.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 -- 172.21.15.186:0/4181790533 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f572406c530 msgr2=0x7f572406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f572406c530 0x7f572406e9f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f573c009b40 tx=0x7f573c009190 comp rx=0 tx=0).stop 2024-03-28T15:01:49.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 -- 172.21.15.186:0/4181790533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57401512b0 msgr2=0x7f574014aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57401512b0 0x7f574014aa00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f573802c620 tx=0x7f5738004a40 comp rx=0 tx=0).stop 2024-03-28T15:01:49.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 -- 172.21.15.186:0/4181790533 shutdown_connections 2024-03-28T15:01:49.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f572406c530 0x7f572406e9f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57401512b0 0x7f574014aa00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5740150ed0 0x7f57401ed7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 --2- 172.21.15.186:0/4181790533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57400733f0 0x7f57401ecdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 -- 172.21.15.186:0/4181790533 >> 172.21.15.186:0/4181790533 conn(0x7f57400c8a20 msgr2=0x7f57400cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:49.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 -- 172.21.15.186:0/4181790533 shutdown_connections 2024-03-28T15:01:49.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.639+0000 7f5748379640 1 -- 172.21.15.186:0/4181790533 wait complete. 2024-03-28T15:01:49.650 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:01:18.661274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:01:29.512445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:01:30.388340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:01:49.650 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:01:49.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.755+0000 7f66131f7640 1 Processor -- start 2024-03-28T15:01:49.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.755+0000 7f66131f7640 1 -- start start 2024-03-28T15:01:49.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f66131f7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f660c15e620 0x7f660c06b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f66131f7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f660c070dd0 0x7f660c165240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f66131f7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f660c06f710 con 0x7f660c157fb0 2024-03-28T15:01:49.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f66131f7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f660c15ee00 con 0x7f660c070dd0 2024-03-28T15:01:49.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f66131f7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f660c0b3560 con 0x7f660c15e620 2024-03-28T15:01:49.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f6611794640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f660c070dd0 0x7f660c165240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f6610f93640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f660c157fb0 0x7f660c15df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49742/0 (socket says 172.21.15.186:49742) 2024-03-28T15:01:49.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f6610f93640 1 -- 172.21.15.186:0/2858313751 learned_addr learned my addr 172.21.15.186:0/2858313751 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:49.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f660bfff640 1 --2- 172.21.15.186:0/2858313751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f660c15e620 0x7f660c06b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f6611794640 1 -- 172.21.15.186:0/2858313751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f660c15e620 msgr2=0x7f660c06b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:49.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f6611794640 1 --2- 172.21.15.186:0/2858313751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f660c15e620 0x7f660c06b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f6611794640 1 -- 172.21.15.186:0/2858313751 >> v1:172.21.15.186:6789/0 conn(0x7f660c157fb0 legacy=0x7f660c15df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f6611794640 1 -- 172.21.15.186:0/2858313751 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c15eb20 con 0x7f660c070dd0 2024-03-28T15:01:49.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f660b7fe640 1 -- 172.21.15.186:0/2858313751 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3957027155 0 0) 0x7f660c06f710 con 0x7f660c157fb0 2024-03-28T15:01:49.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.756+0000 7f660bfff640 1 --2- 172.21.15.186:0/2858313751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f660c15e620 0x7f660c06b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:49.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.757+0000 7f6611794640 1 --2- 172.21.15.186:0/2858313751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f660c070dd0 0x7f660c165240 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f660c06f8e0 tx=0x7f660402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99f0c2a6b37058f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:49.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.757+0000 7f660b7fe640 1 -- 172.21.15.186:0/2858313751 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604039070 con 0x7f660c070dd0 2024-03-28T15:01:49.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.757+0000 7f660b7fe640 1 -- 172.21.15.186:0/2858313751 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660402cc40 con 0x7f660c070dd0 2024-03-28T15:01:49.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.757+0000 7f660b7fe640 1 -- 172.21.15.186:0/2858313751 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604034440 con 0x7f660c070dd0 2024-03-28T15:01:49.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.757+0000 7f66131f7640 1 -- 172.21.15.186:0/2858313751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f660c070dd0 msgr2=0x7f660c165240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.757+0000 7f66131f7640 1 --2- 172.21.15.186:0/2858313751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f660c070dd0 0x7f660c165240 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f660c06f8e0 tx=0x7f660402be90 comp rx=0 tx=0).stop 2024-03-28T15:01:49.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.758+0000 7f66131f7640 1 -- 172.21.15.186:0/2858313751 shutdown_connections 2024-03-28T15:01:49.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.758+0000 7f66131f7640 1 --2- 172.21.15.186:0/2858313751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f660c15e620 0x7f660c06b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.758+0000 7f66131f7640 1 --2- 172.21.15.186:0/2858313751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f660c070dd0 0x7f660c165240 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.758+0000 7f66131f7640 1 -- 172.21.15.186:0/2858313751 >> 172.21.15.186:0/2858313751 conn(0x7f660c073610 msgr2=0x7f660c06d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:49.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.758+0000 7f66131f7640 1 -- 172.21.15.186:0/2858313751 shutdown_connections 2024-03-28T15:01:49.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.758+0000 7f66131f7640 1 -- 172.21.15.186:0/2858313751 wait complete. 2024-03-28T15:01:49.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.759+0000 7f66131f7640 1 Processor -- start 2024-03-28T15:01:49.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.759+0000 7f66131f7640 1 -- start start 2024-03-28T15:01:49.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f66131f7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f660c070dd0 0x7f660c14f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f66131f7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f660c157fb0 0x7f660c0cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f66131f7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f660c15e620 0x7f660c14eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f66131f7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f660c070820 con 0x7f660c157fb0 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f66131f7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f660c06f9b0 con 0x7f660c15e620 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f66131f7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f660c158390 con 0x7f660c070dd0 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f660bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f660c157fb0 0x7f660c0cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f660bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f660c157fb0 0x7f660c0cd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42324/0 (socket says 172.21.15.186:42324) 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f660bfff640 1 -- 172.21.15.186:0/960334156 learned_addr learned my addr 172.21.15.186:0/960334156 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f6611794640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f660c15e620 0x7f660c14eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f6610f93640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f660c070dd0 0x7f660c14f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f660bfff640 1 -- 172.21.15.186:0/960334156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f660c070dd0 msgr2=0x7f660c14f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f660bfff640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f660c070dd0 0x7f660c14f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f660bfff640 1 -- 172.21.15.186:0/960334156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f660c15e620 msgr2=0x7f660c14eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f660bfff640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f660c15e620 0x7f660c14eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f660bfff640 1 -- 172.21.15.186:0/960334156 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c15bf20 con 0x7f660c157fb0 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f6611794640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f660c15e620 0x7f660c14eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f660bfff640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f660c157fb0 0x7f660c0cd120 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f65f800ebe0 tx=0x7f65f800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f66097fa640 1 -- 172.21.15.186:0/960334156 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8017070 con 0x7f660c157fb0 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f66097fa640 1 -- 172.21.15.186:0/960334156 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f800cc30 con 0x7f660c157fb0 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f66131f7640 1 -- 172.21.15.186:0/960334156 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f660c14ddd0 con 0x7f660c157fb0 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f66097fa640 1 -- 172.21.15.186:0/960334156 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8013470 con 0x7f660c157fb0 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.760+0000 7f66131f7640 1 -- 172.21.15.186:0/960334156 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f660c14e390 con 0x7f660c157fb0 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.761+0000 7f66131f7640 1 -- 172.21.15.186:0/960334156 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f660c0681e0 con 0x7f660c157fb0 2024-03-28T15:01:49.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.762+0000 7f66097fa640 1 -- 172.21.15.186:0/960334156 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f65f801a070 con 0x7f660c157fb0 2024-03-28T15:01:49.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.762+0000 7f66097fa640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f65f006c530 0x7f65f006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:49.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.762+0000 7f6610f93640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f65f006c530 0x7f65f006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:49.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.762+0000 7f66097fa640 1 -- 172.21.15.186:0/960334156 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f65f808b550 con 0x7f660c157fb0 2024-03-28T15:01:49.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.763+0000 7f6610f93640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f65f006c530 0x7f65f006e9f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6600000f60 tx=0x7f6600007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:49.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.767+0000 7f66097fa640 1 -- 172.21.15.186:0/960334156 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f65f8059520 con 0x7f660c157fb0 2024-03-28T15:01:49.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.940+0000 7f66131f7640 1 -- 172.21.15.186:0/960334156 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f660c069570 con 0x7f660c157fb0 2024-03-28T15:01:49.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.941+0000 7f66097fa640 1 -- 172.21.15.186:0/960334156 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f65f805d820 con 0x7f660c157fb0 2024-03-28T15:01:49.941 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 1 2024-03-28T15:01:49.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.942+0000 7f66131f7640 1 -- 172.21.15.186:0/960334156 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f65f006c530 msgr2=0x7f65f006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.942+0000 7f66131f7640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f65f006c530 0x7f65f006e9f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6600000f60 tx=0x7f6600007d50 comp rx=0 tx=0).stop 2024-03-28T15:01:49.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.942+0000 7f66131f7640 1 -- 172.21.15.186:0/960334156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f660c157fb0 msgr2=0x7f660c0cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:49.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.942+0000 7f66131f7640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f660c157fb0 0x7f660c0cd120 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f65f800ebe0 tx=0x7f65f800c520 comp rx=0 tx=0).stop 2024-03-28T15:01:49.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.943+0000 7f66131f7640 1 -- 172.21.15.186:0/960334156 shutdown_connections 2024-03-28T15:01:49.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.943+0000 7f66131f7640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f65f006c530 0x7f65f006e9f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.943+0000 7f66131f7640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f660c15e620 0x7f660c14eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.943+0000 7f66131f7640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f660c157fb0 0x7f660c0cd120 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.943+0000 7f66131f7640 1 --2- 172.21.15.186:0/960334156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f660c070dd0 0x7f660c14f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:49.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.943+0000 7f66131f7640 1 -- 172.21.15.186:0/960334156 >> 172.21.15.186:0/960334156 conn(0x7f660c073610 msgr2=0x7f660c06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:49.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.943+0000 7f66131f7640 1 -- 172.21.15.186:0/960334156 shutdown_connections 2024-03-28T15:01:49.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:49.943+0000 7f66131f7640 1 -- 172.21.15.186:0/960334156 wait complete. 2024-03-28T15:01:49.954 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:01:50.057 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.057+0000 7f9f4eadf640 1 Processor -- start 2024-03-28T15:01:50.057 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.057+0000 7f9f4eadf640 1 -- start start 2024-03-28T15:01:50.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f4eadf640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9f4806db50 0x7f9f4806df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f4eadf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f480c8600 con 0x7f9f480c8110 2024-03-28T15:01:50.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f4eadf640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f48157740 con 0x7f9f48151870 2024-03-28T15:01:50.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f4eadf640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f9f48158070 con 0x7f9f4806db50 2024-03-28T15:01:50.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f47fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f9f480c8110 0x7f9f4806ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49754/0 (socket says 172.21.15.186:49754) 2024-03-28T15:01:50.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f47fff640 1 -- 172.21.15.186:0/952565888 learned_addr learned my addr 172.21.15.186:0/952565888 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:50.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f4d07c640 1 --2- 172.21.15.186:0/952565888 >> v2:172.21.15.191:3300/0 conn(0x7f9f4806db50 0x7f9f4806df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f477fe640 1 -- 172.21.15.186:0/952565888 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 431421668 0 0) 0x7f9f480c8600 con 0x7f9f480c8110 2024-03-28T15:01:50.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f477fe640 1 -- 172.21.15.186:0/952565888 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f30003520 con 0x7f9f480c8110 2024-03-28T15:01:50.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f4d07c640 1 -- 172.21.15.186:0/952565888 >> v1:172.21.15.191:6789/0 conn(0x7f9f48151870 legacy=0x7f9f480c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f4d07c640 1 -- 172.21.15.186:0/952565888 >> v1:172.21.15.186:6789/0 conn(0x7f9f480c8110 legacy=0x7f9f4806ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.058+0000 7f9f4d07c640 1 -- 172.21.15.186:0/952565888 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f48157a20 con 0x7f9f4806db50 2024-03-28T15:01:50.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.059+0000 7f9f4d07c640 1 --2- 172.21.15.186:0/952565888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f4806db50 0x7f9f4806df90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009d90 tx=0x7f9f4002bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88cfcdbbe25b0439 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:50.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.059+0000 7f9f477fe640 1 -- 172.21.15.186:0/952565888 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40039070 con 0x7f9f4806db50 2024-03-28T15:01:50.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.059+0000 7f9f477fe640 1 -- 172.21.15.186:0/952565888 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f4002cd80 con 0x7f9f4806db50 2024-03-28T15:01:50.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.059+0000 7f9f477fe640 1 -- 172.21.15.186:0/952565888 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40034640 con 0x7f9f4806db50 2024-03-28T15:01:50.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.060+0000 7f9f4eadf640 1 -- 172.21.15.186:0/952565888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f4806db50 msgr2=0x7f9f4806df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.060+0000 7f9f4eadf640 1 --2- 172.21.15.186:0/952565888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f4806db50 0x7f9f4806df90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009d90 tx=0x7f9f4002bfd0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.060+0000 7f9f4eadf640 1 -- 172.21.15.186:0/952565888 shutdown_connections 2024-03-28T15:01:50.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.060+0000 7f9f4eadf640 1 --2- 172.21.15.186:0/952565888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f4806db50 0x7f9f4806df90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.060+0000 7f9f4eadf640 1 -- 172.21.15.186:0/952565888 >> 172.21.15.186:0/952565888 conn(0x7f9f4806c010 msgr2=0x7f9f480cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:50.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.060+0000 7f9f4eadf640 1 -- 172.21.15.186:0/952565888 shutdown_connections 2024-03-28T15:01:50.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.060+0000 7f9f4eadf640 1 -- 172.21.15.186:0/952565888 wait complete. 2024-03-28T15:01:50.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.061+0000 7f9f4eadf640 1 Processor -- start 2024-03-28T15:01:50.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.061+0000 7f9f4eadf640 1 -- start start 2024-03-28T15:01:50.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4eadf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f4806db50 0x7f9f48135910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4eadf640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f480c8110 0x7f9f48135e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4eadf640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f48151870 0x7f9f480c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4eadf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4806f2f0 con 0x7f9f4806db50 2024-03-28T15:01:50.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4eadf640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9f48151c50 con 0x7f9f480c8110 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4eadf640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f9f48136c40 con 0x7f9f48151870 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4c87b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f4806db50 0x7f9f48135910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4c87b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f4806db50 0x7f9f48135910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42330/0 (socket says 172.21.15.186:42330) 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4c87b640 1 -- 172.21.15.186:0/2058647871 learned_addr learned my addr 172.21.15.186:0/2058647871 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f47fff640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f480c8110 0x7f9f48135e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4d07c640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f48151870 0x7f9f480c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4c87b640 1 -- 172.21.15.186:0/2058647871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f48151870 msgr2=0x7f9f480c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4c87b640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f48151870 0x7f9f480c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4c87b640 1 -- 172.21.15.186:0/2058647871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f480c8110 msgr2=0x7f9f48135e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4c87b640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f480c8110 0x7f9f48135e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4c87b640 1 -- 172.21.15.186:0/2058647871 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f481f1ce0 con 0x7f9f4806db50 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.062+0000 7f9f4c87b640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f4806db50 0x7f9f48135910 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800c3f0 tx=0x7f9f3800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.063+0000 7f9f457fa640 1 -- 172.21.15.186:0/2058647871 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f380052d0 con 0x7f9f4806db50 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.063+0000 7f9f457fa640 1 -- 172.21.15.186:0/2058647871 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f38005490 con 0x7f9f4806db50 2024-03-28T15:01:50.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.063+0000 7f9f457fa640 1 -- 172.21.15.186:0/2058647871 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f38007260 con 0x7f9f4806db50 2024-03-28T15:01:50.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.063+0000 7f9f4eadf640 1 -- 172.21.15.186:0/2058647871 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f481f1fc0 con 0x7f9f4806db50 2024-03-28T15:01:50.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.063+0000 7f9f4eadf640 1 -- 172.21.15.186:0/2058647871 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f480cba10 con 0x7f9f4806db50 2024-03-28T15:01:50.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.064+0000 7f9f4eadf640 1 -- 172.21.15.186:0/2058647871 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f48140e20 con 0x7f9f4806db50 2024-03-28T15:01:50.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.064+0000 7f9f457fa640 1 -- 172.21.15.186:0/2058647871 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9f38007400 con 0x7f9f4806db50 2024-03-28T15:01:50.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.064+0000 7f9f457fa640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9f3406c530 0x7f9f3406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.065+0000 7f9f457fa640 1 -- 172.21.15.186:0/2058647871 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9f3808ab00 con 0x7f9f4806db50 2024-03-28T15:01:50.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.065+0000 7f9f47fff640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9f3406c530 0x7f9f3406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.065+0000 7f9f47fff640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9f3406c530 0x7f9f3406e9f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9f4814e220 tx=0x7f9f3c007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:50.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.069+0000 7f9f457fa640 1 -- 172.21.15.186:0/2058647871 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f9f38058ad0 con 0x7f9f4806db50 2024-03-28T15:01:50.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.245+0000 7f9f4eadf640 1 -- 172.21.15.186:0/2058647871 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9f480cbc10 con 0x7f9f4806db50 2024-03-28T15:01:50.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.245+0000 7f9f457fa640 1 -- 172.21.15.186:0/2058647871 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f9f480cbc10 con 0x7f9f4806db50 2024-03-28T15:01:50.245 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 32 2024-03-28T15:01:50.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 -- 172.21.15.186:0/2058647871 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9f3406c530 msgr2=0x7f9f3406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9f3406c530 0x7f9f3406e9f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9f4814e220 tx=0x7f9f3c007d30 comp rx=0 tx=0).stop 2024-03-28T15:01:50.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 -- 172.21.15.186:0/2058647871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f4806db50 msgr2=0x7f9f48135910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f4806db50 0x7f9f48135910 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800c3f0 tx=0x7f9f3800c8b0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 -- 172.21.15.186:0/2058647871 shutdown_connections 2024-03-28T15:01:50.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9f3406c530 0x7f9f3406e9f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f48151870 0x7f9f480c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f480c8110 0x7f9f48135e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 --2- 172.21.15.186:0/2058647871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f4806db50 0x7f9f48135910 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 -- 172.21.15.186:0/2058647871 >> 172.21.15.186:0/2058647871 conn(0x7f9f4806c010 msgr2=0x7f9f4806cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:50.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 -- 172.21.15.186:0/2058647871 shutdown_connections 2024-03-28T15:01:50.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.247+0000 7f9f4eadf640 1 -- 172.21.15.186:0/2058647871 wait complete. 2024-03-28T15:01:50.258 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:01:50.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.361+0000 7f68ddf81640 1 Processor -- start 2024-03-28T15:01:50.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.361+0000 7f68ddf81640 1 -- start start 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68ddf81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8157fb0 0x7f68d806b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68ddf81640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f68d8070dd0 0x7f68d815df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68ddf81640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f68d80c89b0 0x7f68d815e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68ddf81640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f68d815ee00 con 0x7f68d80c89b0 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68ddf81640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f68d806f710 con 0x7f68d8070dd0 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68ddf81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f68d80b3560 con 0x7f68d8157fb0 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68d7fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f68d80c89b0 0x7f68d815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68d7fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f68d80c89b0 0x7f68d815e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42336/0 (socket says 172.21.15.186:42336) 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68d77fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8157fb0 0x7f68d806b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68d7fff640 1 -- 172.21.15.186:0/663593477 learned_addr learned my addr 172.21.15.186:0/663593477 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68d6ffd640 1 --2- 172.21.15.186:0/663593477 >> v2:172.21.15.191:3300/0 conn(0x7f68d8070dd0 0x7f68d815df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68d7fff640 1 -- 172.21.15.186:0/663593477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8157fb0 msgr2=0x7f68d806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:50.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68d7fff640 1 --2- 172.21.15.186:0/663593477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8157fb0 0x7f68d806b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.362+0000 7f68d7fff640 1 -- 172.21.15.186:0/663593477 >> v2:172.21.15.191:3300/0 conn(0x7f68d8070dd0 msgr2=0x7f68d815df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:50.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.363+0000 7f68d7fff640 1 --2- 172.21.15.186:0/663593477 >> v2:172.21.15.191:3300/0 conn(0x7f68d8070dd0 0x7f68d815df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.363+0000 7f68d7fff640 1 -- 172.21.15.186:0/663593477 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d815eb20 con 0x7f68d80c89b0 2024-03-28T15:01:50.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.363+0000 7f68d6ffd640 1 --2- 172.21.15.186:0/663593477 >> v2:172.21.15.191:3300/0 conn(0x7f68d8070dd0 0x7f68d815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:50.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.363+0000 7f68d77fe640 1 --2- 172.21.15.186:0/663593477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8157fb0 0x7f68d806b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:50.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.363+0000 7f68d7fff640 1 --2- 172.21.15.186:0/663593477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d80c89b0 0x7f68d815e450 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f68d8073fa0 tx=0x7f68d002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ee920eb3b765c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:50.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.363+0000 7f68d67fc640 1 -- 172.21.15.186:0/663593477 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d002ca70 con 0x7f68d80c89b0 2024-03-28T15:01:50.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.363+0000 7f68d67fc640 1 -- 172.21.15.186:0/663593477 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68d002cc10 con 0x7f68d80c89b0 2024-03-28T15:01:50.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.363+0000 7f68d67fc640 1 -- 172.21.15.186:0/663593477 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d00343f0 con 0x7f68d80c89b0 2024-03-28T15:01:50.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.364+0000 7f68ddf81640 1 -- 172.21.15.186:0/663593477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d80c89b0 msgr2=0x7f68d815e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.364+0000 7f68ddf81640 1 --2- 172.21.15.186:0/663593477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d80c89b0 0x7f68d815e450 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f68d8073fa0 tx=0x7f68d002bed0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.364+0000 7f68ddf81640 1 -- 172.21.15.186:0/663593477 shutdown_connections 2024-03-28T15:01:50.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.364+0000 7f68ddf81640 1 --2- 172.21.15.186:0/663593477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d80c89b0 0x7f68d815e450 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.364+0000 7f68ddf81640 1 --2- 172.21.15.186:0/663593477 >> v2:172.21.15.191:3300/0 conn(0x7f68d8070dd0 0x7f68d815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.364+0000 7f68ddf81640 1 --2- 172.21.15.186:0/663593477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8157fb0 0x7f68d806b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.364+0000 7f68ddf81640 1 -- 172.21.15.186:0/663593477 >> 172.21.15.186:0/663593477 conn(0x7f68d8073610 msgr2=0x7f68d806d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:50.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.364+0000 7f68ddf81640 1 -- 172.21.15.186:0/663593477 shutdown_connections 2024-03-28T15:01:50.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.364+0000 7f68ddf81640 1 -- 172.21.15.186:0/663593477 wait complete. 2024-03-28T15:01:50.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.365+0000 7f68ddf81640 1 Processor -- start 2024-03-28T15:01:50.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.365+0000 7f68ddf81640 1 -- start start 2024-03-28T15:01:50.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.366+0000 7f68ddf81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8070dd0 0x7f68d8154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.366+0000 7f68ddf81640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d80c89b0 0x7f68d8156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.366+0000 7f68ddf81640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d8157fb0 0x7f68d81570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.366+0000 7f68ddf81640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f68d8070820 con 0x7f68d8157fb0 2024-03-28T15:01:50.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.366+0000 7f68ddf81640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f68d806f9b0 con 0x7f68d80c89b0 2024-03-28T15:01:50.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.366+0000 7f68ddf81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f68d8158390 con 0x7f68d8070dd0 2024-03-28T15:01:50.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.366+0000 7f68d7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d8157fb0 0x7f68d81570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.366+0000 7f68d7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d8157fb0 0x7f68d81570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42352/0 (socket says 172.21.15.186:42352) 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.366+0000 7f68d77fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8070dd0 0x7f68d8154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.366+0000 7f68d6ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d80c89b0 0x7f68d8156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.366+0000 7f68d7fff640 1 -- 172.21.15.186:0/3090345265 learned_addr learned my addr 172.21.15.186:0/3090345265 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68d77fe640 1 -- 172.21.15.186:0/3090345265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d80c89b0 msgr2=0x7f68d8156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68d77fe640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d80c89b0 0x7f68d8156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68d77fe640 1 -- 172.21.15.186:0/3090345265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d8157fb0 msgr2=0x7f68d81570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68d77fe640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d8157fb0 0x7f68d81570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68d77fe640 1 -- 172.21.15.186:0/3090345265 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d8154680 con 0x7f68d8070dd0 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68d6ffd640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d80c89b0 0x7f68d8156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68d7fff640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d8157fb0 0x7f68d81570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68d77fe640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8070dd0 0x7f68d8154140 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68c800e900 tx=0x7f68c800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68bffff640 1 -- 172.21.15.186:0/3090345265 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c8017070 con 0x7f68d8070dd0 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68bffff640 1 -- 172.21.15.186:0/3090345265 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c800ca00 con 0x7f68d8070dd0 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68ddf81640 1 -- 172.21.15.186:0/3090345265 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d815d310 con 0x7f68d8070dd0 2024-03-28T15:01:50.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68bffff640 1 -- 172.21.15.186:0/3090345265 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c800cdf0 con 0x7f68d8070dd0 2024-03-28T15:01:50.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.367+0000 7f68ddf81640 1 -- 172.21.15.186:0/3090345265 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d815d8d0 con 0x7f68d8070dd0 2024-03-28T15:01:50.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.368+0000 7f68ddf81640 1 -- 172.21.15.186:0/3090345265 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d80681e0 con 0x7f68d8070dd0 2024-03-28T15:01:50.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.368+0000 7f68bffff640 1 -- 172.21.15.186:0/3090345265 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f68c801a070 con 0x7f68d8070dd0 2024-03-28T15:01:50.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.369+0000 7f68bffff640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68c406c530 0x7f68c406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.369+0000 7f68bffff640 1 -- 172.21.15.186:0/3090345265 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f68c808b4a0 con 0x7f68d8070dd0 2024-03-28T15:01:50.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.369+0000 7f68d6ffd640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68c406c530 0x7f68c406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.369+0000 7f68d6ffd640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68c406c530 0x7f68c406e9f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009580 tx=0x7f68cc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:50.373 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.373+0000 7f68bffff640 1 -- 172.21.15.186:0/3090345265 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f68c8059450 con 0x7f68d8070dd0 2024-03-28T15:01:50.547 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.547+0000 7f68ddf81640 1 -- 172.21.15.186:0/3090345265 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f68d81577d0 con 0x7f68d8070dd0 2024-03-28T15:01:50.547 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.547+0000 7f68bffff640 1 -- 172.21.15.186:0/3090345265 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f68c805d750 con 0x7f68d8070dd0 2024-03-28T15:01:50.548 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 64 2024-03-28T15:01:50.549 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 -- 172.21.15.186:0/3090345265 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68c406c530 msgr2=0x7f68c406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68c406c530 0x7f68c406e9f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009580 tx=0x7f68cc009190 comp rx=0 tx=0).stop 2024-03-28T15:01:50.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 -- 172.21.15.186:0/3090345265 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8070dd0 msgr2=0x7f68d8154140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8070dd0 0x7f68d8154140 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68c800e900 tx=0x7f68c800edc0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 -- 172.21.15.186:0/3090345265 shutdown_connections 2024-03-28T15:01:50.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68c406c530 0x7f68c406e9f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d8157fb0 0x7f68d81570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d80c89b0 0x7f68d8156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 --2- 172.21.15.186:0/3090345265 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d8070dd0 0x7f68d8154140 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 -- 172.21.15.186:0/3090345265 >> 172.21.15.186:0/3090345265 conn(0x7f68d8073610 msgr2=0x7f68d80c9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:50.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 -- 172.21.15.186:0/3090345265 shutdown_connections 2024-03-28T15:01:50.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.549+0000 7f68ddf81640 1 -- 172.21.15.186:0/3090345265 wait complete. 2024-03-28T15:01:50.560 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-03-28T15:01:50.560 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:01:50.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.661+0000 7fe4673f7640 1 Processor -- start 2024-03-28T15:01:50.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.661+0000 7fe4673f7640 1 -- start start 2024-03-28T15:01:50.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe4673f7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4601512b0 0x7fe4600c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe4673f7640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe4600733f0 0x7fe4600b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe4673f7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4600c1b40 con 0x7fe460150ed0 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe4673f7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fe4600c1e20 con 0x7fe4601512b0 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe4673f7640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fe4600c18a0 con 0x7fe4600733f0 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe464992640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4601512b0 0x7fe4600c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe465193640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe460150ed0 0x7fe460156d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49762/0 (socket says 172.21.15.186:49762) 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe464992640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4601512b0 0x7fe4600c0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42364/0 (socket says 172.21.15.186:42364) 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe465193640 1 -- 172.21.15.186:0/2885084079 learned_addr learned my addr 172.21.15.186:0/2885084079 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe465994640 1 --2- 172.21.15.186:0/2885084079 >> v2:172.21.15.191:3300/0 conn(0x7fe4600733f0 0x7fe4600b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe464992640 1 -- 172.21.15.186:0/2885084079 >> v2:172.21.15.191:3300/0 conn(0x7fe4600733f0 msgr2=0x7fe4600b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe464992640 1 --2- 172.21.15.186:0/2885084079 >> v2:172.21.15.191:3300/0 conn(0x7fe4600733f0 0x7fe4600b1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe464992640 1 -- 172.21.15.186:0/2885084079 >> v1:172.21.15.186:6789/0 conn(0x7fe460150ed0 legacy=0x7fe460156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe457fff640 1 -- 172.21.15.186:0/2885084079 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3972154172 0 0) 0x7fe4600c1b40 con 0x7fe460150ed0 2024-03-28T15:01:50.664 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.662+0000 7fe464992640 1 -- 172.21.15.186:0/2885084079 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4600c15c0 con 0x7fe4601512b0 2024-03-28T15:01:50.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.663+0000 7fe464992640 1 --2- 172.21.15.186:0/2885084079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4601512b0 0x7fe4600c0f40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe460067cd0 tx=0x7fe45002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d07a2986c6ac1aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:50.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.663+0000 7fe457fff640 1 -- 172.21.15.186:0/2885084079 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450034070 con 0x7fe4601512b0 2024-03-28T15:01:50.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.663+0000 7fe457fff640 1 -- 172.21.15.186:0/2885084079 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe450030610 con 0x7fe4601512b0 2024-03-28T15:01:50.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.663+0000 7fe457fff640 1 -- 172.21.15.186:0/2885084079 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4500309e0 con 0x7fe4601512b0 2024-03-28T15:01:50.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.663+0000 7fe4673f7640 1 -- 172.21.15.186:0/2885084079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4601512b0 msgr2=0x7fe4600c0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.663+0000 7fe4673f7640 1 --2- 172.21.15.186:0/2885084079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4601512b0 0x7fe4600c0f40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe460067cd0 tx=0x7fe45002b960 comp rx=0 tx=0).stop 2024-03-28T15:01:50.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.664+0000 7fe4673f7640 1 -- 172.21.15.186:0/2885084079 shutdown_connections 2024-03-28T15:01:50.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.664+0000 7fe4673f7640 1 --2- 172.21.15.186:0/2885084079 >> v2:172.21.15.191:3300/0 conn(0x7fe4600733f0 0x7fe4600b1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.664+0000 7fe4673f7640 1 --2- 172.21.15.186:0/2885084079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4601512b0 0x7fe4600c0f40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.664+0000 7fe4673f7640 1 -- 172.21.15.186:0/2885084079 >> 172.21.15.186:0/2885084079 conn(0x7fe4600c8a20 msgr2=0x7fe4600c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:50.670 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.664+0000 7fe4673f7640 1 -- 172.21.15.186:0/2885084079 shutdown_connections 2024-03-28T15:01:50.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.664+0000 7fe4673f7640 1 -- 172.21.15.186:0/2885084079 wait complete. 2024-03-28T15:01:50.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.665+0000 7fe4673f7640 1 Processor -- start 2024-03-28T15:01:50.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.665+0000 7fe4673f7640 1 -- start start 2024-03-28T15:01:50.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.665+0000 7fe4673f7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4600733f0 0x7fe4601f08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.665+0000 7fe4673f7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe460150ed0 0x7fe4601615f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe4673f7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4601512b0 0x7fe460141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe4673f7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe46006e5b0 con 0x7fe4600733f0 2024-03-28T15:01:50.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe4673f7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe460072f60 con 0x7fe4601512b0 2024-03-28T15:01:50.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe4673f7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fe4600712b0 con 0x7fe460150ed0 2024-03-28T15:01:50.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe465193640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4600733f0 0x7fe4601f08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe465193640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4600733f0 0x7fe4601f08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42366/0 (socket says 172.21.15.186:42366) 2024-03-28T15:01:50.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe465193640 1 -- 172.21.15.186:0/4179833790 learned_addr learned my addr 172.21.15.186:0/4179833790 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:50.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe464992640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe460150ed0 0x7fe4601615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe465994640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4601512b0 0x7fe460141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe465193640 1 -- 172.21.15.186:0/4179833790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe460150ed0 msgr2=0x7fe4601615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe465193640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe460150ed0 0x7fe4601615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe465193640 1 -- 172.21.15.186:0/4179833790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4601512b0 msgr2=0x7fe460141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe465193640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4601512b0 0x7fe460141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe465193640 1 -- 172.21.15.186:0/4179833790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe460161b30 con 0x7fe4600733f0 2024-03-28T15:01:50.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe465193640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4600733f0 0x7fe4601f08b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe45c00c500 tx=0x7fe45c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:50.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe455ffb640 1 -- 172.21.15.186:0/4179833790 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c018070 con 0x7fe4600733f0 2024-03-28T15:01:50.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe455ffb640 1 -- 172.21.15.186:0/4179833790 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe45c007580 con 0x7fe4600733f0 2024-03-28T15:01:50.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe4673f7640 1 -- 172.21.15.186:0/4179833790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4601423b0 con 0x7fe4600733f0 2024-03-28T15:01:50.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe455ffb640 1 -- 172.21.15.186:0/4179833790 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c004a20 con 0x7fe4600733f0 2024-03-28T15:01:50.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.666+0000 7fe4673f7640 1 -- 172.21.15.186:0/4179833790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4600c5ef0 con 0x7fe4600733f0 2024-03-28T15:01:50.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.668+0000 7fe4673f7640 1 -- 172.21.15.186:0/4179833790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe460069500 con 0x7fe4600733f0 2024-03-28T15:01:50.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.668+0000 7fe455ffb640 1 -- 172.21.15.186:0/4179833790 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe45c01b070 con 0x7fe4600733f0 2024-03-28T15:01:50.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.668+0000 7fe455ffb640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe44406c530 0x7fe44406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:50.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.668+0000 7fe455ffb640 1 -- 172.21.15.186:0/4179833790 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe45c08a930 con 0x7fe4600733f0 2024-03-28T15:01:50.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.668+0000 7fe464992640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe44406c530 0x7fe44406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:50.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.669+0000 7fe464992640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe44406c530 0x7fe44406e9f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe450036000 tx=0x7fe450008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:50.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.673+0000 7fe455ffb640 1 -- 172.21.15.186:0/4179833790 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fe45c058900 con 0x7fe4600733f0 2024-03-28T15:01:50.888 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.888+0000 7fe4673f7640 1 -- 172.21.15.186:0/4179833790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4600c81e0 con 0x7fe4600733f0 2024-03-28T15:01:50.889 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.889+0000 7fe455ffb640 1 -- 172.21.15.186:0/4179833790 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7fe45c05cc00 con 0x7fe4600733f0 2024-03-28T15:01:50.889 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:50.889 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:01:50.890 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:01:50.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 -- 172.21.15.186:0/4179833790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe44406c530 msgr2=0x7fe44406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe44406c530 0x7fe44406e9f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe450036000 tx=0x7fe450008030 comp rx=0 tx=0).stop 2024-03-28T15:01:50.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 -- 172.21.15.186:0/4179833790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4600733f0 msgr2=0x7fe4601f08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:50.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4600733f0 0x7fe4601f08b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe45c00c500 tx=0x7fe45c00c9c0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 -- 172.21.15.186:0/4179833790 shutdown_connections 2024-03-28T15:01:50.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe44406c530 0x7fe44406e9f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4601512b0 0x7fe460141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe460150ed0 0x7fe4601615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 --2- 172.21.15.186:0/4179833790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4600733f0 0x7fe4601f08b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:50.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 -- 172.21.15.186:0/4179833790 >> 172.21.15.186:0/4179833790 conn(0x7fe4600c8a20 msgr2=0x7fe46014ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:50.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 -- 172.21.15.186:0/4179833790 shutdown_connections 2024-03-28T15:01:50.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:50.891+0000 7fe4673f7640 1 -- 172.21.15.186:0/4179833790 wait complete. 2024-03-28T15:01:50.902 INFO:tasks.mds_thrash:Ready to start thrashing 2024-03-28T15:01:50.902 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-03-28T15:01:50.902 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:01:51.004 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.004+0000 7ff7760c9640 1 Processor -- start 2024-03-28T15:01:51.004 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.004+0000 7ff7760c9640 1 -- start start 2024-03-28T15:01:51.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.004+0000 7ff7760c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff7700caa70 0x7ff7700c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.004+0000 7ff7760c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7700c6a00 0x7ff7700cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.004+0000 7ff7760c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7700c6f90 con 0x7ff7700c5380 2024-03-28T15:01:51.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.004+0000 7ff7760c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7ff770071a80 con 0x7ff7700caa70 2024-03-28T15:01:51.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.004+0000 7ff7760c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff7700c6030 con 0x7ff7700c6a00 2024-03-28T15:01:51.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.005+0000 7ff76ffff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff7700c5380 0x7ff7700cb390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49770/0 (socket says 172.21.15.186:49770) 2024-03-28T15:01:51.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.005+0000 7ff76ffff640 1 -- 172.21.15.186:0/920588296 learned_addr learned my addr 172.21.15.186:0/920588296 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:51.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.005+0000 7ff76effd640 1 --2- 172.21.15.186:0/920588296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7700c6a00 0x7ff7700cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.005+0000 7ff76f7fe640 1 --2- 172.21.15.186:0/920588296 >> v2:172.21.15.191:3300/0 conn(0x7ff7700caa70 0x7ff7700c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.005+0000 7ff76effd640 1 -- 172.21.15.186:0/920588296 >> v2:172.21.15.191:3300/0 conn(0x7ff7700caa70 msgr2=0x7ff7700c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:51.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.005+0000 7ff76effd640 1 --2- 172.21.15.186:0/920588296 >> v2:172.21.15.191:3300/0 conn(0x7ff7700caa70 0x7ff7700c64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.005+0000 7ff76e7fc640 1 -- 172.21.15.186:0/920588296 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1437148760 0 0) 0x7ff768001490 con 0x7ff7700c5380 2024-03-28T15:01:51.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.005+0000 7ff76effd640 1 -- 172.21.15.186:0/920588296 >> v1:172.21.15.186:6789/0 conn(0x7ff7700c5380 legacy=0x7ff7700cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.006 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.005+0000 7ff76effd640 1 -- 172.21.15.186:0/920588296 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7700cc970 con 0x7ff7700c6a00 2024-03-28T15:01:51.007 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff76effd640 1 --2- 172.21.15.186:0/920588296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7700c6a00 0x7ff7700cae50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7600097a0 tx=0x7ff76002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a5f57b7e973e67c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff76e7fc640 1 -- 172.21.15.186:0/920588296 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760035020 con 0x7ff7700c6a00 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff76e7fc640 1 -- 172.21.15.186:0/920588296 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff760031830 con 0x7ff7700c6a00 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff76e7fc640 1 -- 172.21.15.186:0/920588296 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760031c00 con 0x7ff7700c6a00 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff7760c9640 1 -- 172.21.15.186:0/920588296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7700c6a00 msgr2=0x7ff7700cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff7760c9640 1 --2- 172.21.15.186:0/920588296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7700c6a00 0x7ff7700cae50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7600097a0 tx=0x7ff76002b8d0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff7760c9640 1 -- 172.21.15.186:0/920588296 shutdown_connections 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff7760c9640 1 --2- 172.21.15.186:0/920588296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7700c6a00 0x7ff7700cae50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff7760c9640 1 --2- 172.21.15.186:0/920588296 >> v2:172.21.15.191:3300/0 conn(0x7ff7700caa70 0x7ff7700c64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff7760c9640 1 -- 172.21.15.186:0/920588296 >> 172.21.15.186:0/920588296 conn(0x7ff7700d00f0 msgr2=0x7ff7700d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff7760c9640 1 -- 172.21.15.186:0/920588296 shutdown_connections 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.006+0000 7ff7760c9640 1 -- 172.21.15.186:0/920588296 wait complete. 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.007+0000 7ff7760c9640 1 Processor -- start 2024-03-28T15:01:51.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff7760c9640 1 -- start start 2024-03-28T15:01:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff7760c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7700c5380 0x7ff7700c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff7760c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7700c6a00 0x7ff7700c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff7760c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7700caa70 0x7ff7700b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff7760c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7700c5cc0 con 0x7ff7700caa70 2024-03-28T15:01:51.010 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff7760c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff77015b530 con 0x7ff7700c5380 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff7760c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff77015b810 con 0x7ff7700c6a00 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff76ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7700caa70 0x7ff7700b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff76f7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7700c5380 0x7ff7700c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff76effd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7700c6a00 0x7ff7700c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff76ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7700caa70 0x7ff7700b5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42374/0 (socket says 172.21.15.186:42374) 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.008+0000 7ff76ffff640 1 -- 172.21.15.186:0/1066968610 learned_addr learned my addr 172.21.15.186:0/1066968610 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.009+0000 7ff76f7fe640 1 -- 172.21.15.186:0/1066968610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7700c6a00 msgr2=0x7ff7700c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.009+0000 7ff76f7fe640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7700c6a00 0x7ff7700c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.009+0000 7ff76f7fe640 1 -- 172.21.15.186:0/1066968610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7700caa70 msgr2=0x7ff7700b5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.009+0000 7ff76f7fe640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7700caa70 0x7ff7700b5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.009+0000 7ff76f7fe640 1 -- 172.21.15.186:0/1066968610 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7700c9e30 con 0x7ff7700c5380 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.009+0000 7ff76f7fe640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7700c5380 0x7ff7700c7f00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff76400e870 tx=0x7ff76400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.009+0000 7ff757fff640 1 -- 172.21.15.186:0/1066968610 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff764017070 con 0x7ff7700c5380 2024-03-28T15:01:51.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.009+0000 7ff757fff640 1 -- 172.21.15.186:0/1066968610 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff76400ee60 con 0x7ff7700c5380 2024-03-28T15:01:51.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.009+0000 7ff757fff640 1 -- 172.21.15.186:0/1066968610 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76400cb60 con 0x7ff7700c5380 2024-03-28T15:01:51.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.009+0000 7ff7760c9640 1 -- 172.21.15.186:0/1066968610 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7700ca0d0 con 0x7ff7700c5380 2024-03-28T15:01:51.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.009+0000 7ff7760c9640 1 -- 172.21.15.186:0/1066968610 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff770157190 con 0x7ff7700c5380 2024-03-28T15:01:51.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.010+0000 7ff7760c9640 1 -- 172.21.15.186:0/1066968610 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7700c4fd0 con 0x7ff7700c5380 2024-03-28T15:01:51.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.010+0000 7ff757fff640 1 -- 172.21.15.186:0/1066968610 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff76401a070 con 0x7ff7700c5380 2024-03-28T15:01:51.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.011+0000 7ff757fff640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff75c06c530 0x7ff75c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.011+0000 7ff757fff640 1 -- 172.21.15.186:0/1066968610 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff76408b550 con 0x7ff7700c5380 2024-03-28T15:01:51.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.011+0000 7ff76effd640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff75c06c530 0x7ff75c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.011+0000 7ff76effd640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff75c06c530 0x7ff75c06e9f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff760036000 tx=0x7ff76003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:51.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.016+0000 7ff757fff640 1 -- 172.21.15.186:0/1066968610 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff7700c4fd0 con 0x7ff7700c5380 2024-03-28T15:01:51.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.193+0000 7ff7760c9640 1 -- 172.21.15.186:0/1066968610 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7700cb820 con 0x7ff75c06c530 2024-03-28T15:01:51.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.199+0000 7ff757fff640 1 -- 172.21.15.186:0/1066968610 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251831 (secure 0 0 0) 0x7ff7600887d0 con 0x7ff75c06c530 2024-03-28T15:01:51.202 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:51.203 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:01:51.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.202+0000 7ff7760c9640 1 -- 172.21.15.186:0/1066968610 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff75c06c530 msgr2=0x7ff75c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.202+0000 7ff7760c9640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff75c06c530 0x7ff75c06e9f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff760036000 tx=0x7ff76003d040 comp rx=0 tx=0).stop 2024-03-28T15:01:51.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.202+0000 7ff7760c9640 1 -- 172.21.15.186:0/1066968610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7700c5380 msgr2=0x7ff7700c7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.202+0000 7ff7760c9640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7700c5380 0x7ff7700c7f00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff76400e870 tx=0x7ff76400ed30 comp rx=0 tx=0).stop 2024-03-28T15:01:51.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.202+0000 7ff7760c9640 1 -- 172.21.15.186:0/1066968610 shutdown_connections 2024-03-28T15:01:51.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.202+0000 7ff7760c9640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff75c06c530 0x7ff75c06e9f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.202+0000 7ff7760c9640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7700caa70 0x7ff7700b5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.202+0000 7ff7760c9640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7700c6a00 0x7ff7700c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.202+0000 7ff7760c9640 1 --2- 172.21.15.186:0/1066968610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7700c5380 0x7ff7700c7f00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.203+0000 7ff7760c9640 1 -- 172.21.15.186:0/1066968610 >> 172.21.15.186:0/1066968610 conn(0x7ff7700d00f0 msgr2=0x7ff770151e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:51.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.203+0000 7ff7760c9640 1 -- 172.21.15.186:0/1066968610 shutdown_connections 2024-03-28T15:01:51.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.203+0000 7ff7760c9640 1 -- 172.21.15.186:0/1066968610 wait complete. 2024-03-28T15:01:51.214 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-28T15:01:50.638678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594034,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218952,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755768,"statfs":{"total":773094113280,"available":772869906432,"internally_reserved":0,"allocated":2891776,"data_stored":1755540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1256,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002386"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461304+0000","last_change":"2024-03-28T15:01:32.122226+0000","last_active":"2024-03-28T15:01:33.461304+0000","last_peered":"2024-03-28T15:01:33.461304+0000","last_clean":"2024-03-28T15:01:33.461304+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:01:33.461304+0000","last_undegraded":"2024-03-28T15:01:33.461304+0000","last_fullsized":"2024-03-28T15:01:33.461304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104984+0000","last_change":"2024-03-28T15:01:32.115850+0000","last_active":"2024-03-28T15:01:33.104984+0000","last_peered":"2024-03-28T15:01:33.104984+0000","last_clean":"2024-03-28T15:01:33.104984+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:01:33.104984+0000","last_undegraded":"2024-03-28T15:01:33.104984+0000","last_fullsized":"2024-03-28T15:01:33.104984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.119744+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:57.022614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107952+0000","last_change":"2024-03-28T15:01:32.120916+0000","last_active":"2024-03-28T15:01:33.107952+0000","last_peered":"2024-03-28T15:01:33.107952+0000","last_clean":"2024-03-28T15:01:33.107952+0000","last_became_active":"2024-03-28T15:01:32.117449+0000","last_became_peered":"2024-03-28T15:01:32.117449+0000","last_unstale":"2024-03-28T15:01:33.107952+0000","last_undegraded":"2024-03-28T15:01:33.107952+0000","last_fullsized":"2024-03-28T15:01:33.107952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461016+0000","last_change":"2024-03-28T15:01:32.112835+0000","last_active":"2024-03-28T15:01:33.461016+0000","last_peered":"2024-03-28T15:01:33.461016+0000","last_clean":"2024-03-28T15:01:33.461016+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:01:33.461016+0000","last_undegraded":"2024-03-28T15:01:33.461016+0000","last_fullsized":"2024-03-28T15:01:33.461016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107714+0000","last_change":"2024-03-28T15:01:32.116248+0000","last_active":"2024-03-28T15:01:33.107714+0000","last_peered":"2024-03-28T15:01:33.107714+0000","last_clean":"2024-03-28T15:01:33.107714+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:01:33.107714+0000","last_undegraded":"2024-03-28T15:01:33.107714+0000","last_fullsized":"2024-03-28T15:01:33.107714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460486+0000","last_change":"2024-03-28T15:01:32.115221+0000","last_active":"2024-03-28T15:01:33.460486+0000","last_peered":"2024-03-28T15:01:33.460486+0000","last_clean":"2024-03-28T15:01:33.460486+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:01:33.460486+0000","last_undegraded":"2024-03-28T15:01:33.460486+0000","last_fullsized":"2024-03-28T15:01:33.460486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460218+0000","last_change":"2024-03-28T15:01:32.118310+0000","last_active":"2024-03-28T15:01:33.460218+0000","last_peered":"2024-03-28T15:01:33.460218+0000","last_clean":"2024-03-28T15:01:33.460218+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:01:33.460218+0000","last_undegraded":"2024-03-28T15:01:33.460218+0000","last_fullsized":"2024-03-28T15:01:33.460218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461212+0000","last_change":"2024-03-28T15:01:32.114205+0000","last_active":"2024-03-28T15:01:33.461212+0000","last_peered":"2024-03-28T15:01:33.461212+0000","last_clean":"2024-03-28T15:01:33.461212+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:01:33.461212+0000","last_undegraded":"2024-03-28T15:01:33.461212+0000","last_fullsized":"2024-03-28T15:01:33.461212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460905+0000","last_change":"2024-03-28T15:01:32.125178+0000","last_active":"2024-03-28T15:01:33.460905+0000","last_peered":"2024-03-28T15:01:33.460905+0000","last_clean":"2024-03-28T15:01:33.460905+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:01:33.460905+0000","last_undegraded":"2024-03-28T15:01:33.460905+0000","last_fullsized":"2024-03-28T15:01:33.460905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460676+0000","last_change":"2024-03-28T15:01:32.125839+0000","last_active":"2024-03-28T15:01:33.460676+0000","last_peered":"2024-03-28T15:01:33.460676+0000","last_clean":"2024-03-28T15:01:33.460676+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:01:33.460676+0000","last_undegraded":"2024-03-28T15:01:33.460676+0000","last_fullsized":"2024-03-28T15:01:33.460676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107231+0000","last_change":"2024-03-28T15:01:32.121874+0000","last_active":"2024-03-28T15:01:33.107231+0000","last_peered":"2024-03-28T15:01:33.107231+0000","last_clean":"2024-03-28T15:01:33.107231+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:01:33.107231+0000","last_undegraded":"2024-03-28T15:01:33.107231+0000","last_fullsized":"2024-03-28T15:01:33.107231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:20.744143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460735+0000","last_change":"2024-03-28T15:01:32.119348+0000","last_active":"2024-03-28T15:01:33.460735+0000","last_peered":"2024-03-28T15:01:33.460735+0000","last_clean":"2024-03-28T15:01:33.460735+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:01:33.460735+0000","last_undegraded":"2024-03-28T15:01:33.460735+0000","last_fullsized":"2024-03-28T15:01:33.460735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461204+0000","last_change":"2024-03-28T15:01:32.114290+0000","last_active":"2024-03-28T15:01:33.461204+0000","last_peered":"2024-03-28T15:01:33.461204+0000","last_clean":"2024-03-28T15:01:33.461204+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:01:33.461204+0000","last_undegraded":"2024-03-28T15:01:33.461204+0000","last_fullsized":"2024-03-28T15:01:33.461204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105071+0000","last_change":"2024-03-28T15:01:32.113654+0000","last_active":"2024-03-28T15:01:33.105071+0000","last_peered":"2024-03-28T15:01:33.105071+0000","last_clean":"2024-03-28T15:01:33.105071+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:01:33.105071+0000","last_undegraded":"2024-03-28T15:01:33.105071+0000","last_fullsized":"2024-03-28T15:01:33.105071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107620+0000","last_change":"2024-03-28T15:01:32.115901+0000","last_active":"2024-03-28T15:01:33.107620+0000","last_peered":"2024-03-28T15:01:33.107620+0000","last_clean":"2024-03-28T15:01:33.107620+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:01:33.107620+0000","last_undegraded":"2024-03-28T15:01:33.107620+0000","last_fullsized":"2024-03-28T15:01:33.107620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108153+0000","last_change":"2024-03-28T15:01:32.113156+0000","last_active":"2024-03-28T15:01:33.108153+0000","last_peered":"2024-03-28T15:01:33.108153+0000","last_clean":"2024-03-28T15:01:33.108153+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:01:33.108153+0000","last_undegraded":"2024-03-28T15:01:33.108153+0000","last_fullsized":"2024-03-28T15:01:33.108153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460557+0000","last_change":"2024-03-28T15:01:32.119785+0000","last_active":"2024-03-28T15:01:33.460557+0000","last_peered":"2024-03-28T15:01:33.460557+0000","last_clean":"2024-03-28T15:01:33.460557+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:01:33.460557+0000","last_undegraded":"2024-03-28T15:01:33.460557+0000","last_fullsized":"2024-03-28T15:01:33.460557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461426+0000","last_change":"2024-03-28T15:01:32.111912+0000","last_active":"2024-03-28T15:01:33.461426+0000","last_peered":"2024-03-28T15:01:33.461426+0000","last_clean":"2024-03-28T15:01:33.461426+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:01:33.461426+0000","last_undegraded":"2024-03-28T15:01:33.461426+0000","last_fullsized":"2024-03-28T15:01:33.461426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:52.632729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461025+0000","last_change":"2024-03-28T15:01:32.124381+0000","last_active":"2024-03-28T15:01:33.461025+0000","last_peered":"2024-03-28T15:01:33.461025+0000","last_clean":"2024-03-28T15:01:33.461025+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:01:33.461025+0000","last_undegraded":"2024-03-28T15:01:33.461025+0000","last_fullsized":"2024-03-28T15:01:33.461025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.120110+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107929+0000","last_change":"2024-03-28T15:01:32.113699+0000","last_active":"2024-03-28T15:01:33.107929+0000","last_peered":"2024-03-28T15:01:33.107929+0000","last_clean":"2024-03-28T15:01:33.107929+0000","last_became_active":"2024-03-28T15:01:32.113321+0000","last_became_peered":"2024-03-28T15:01:32.113321+0000","last_unstale":"2024-03-28T15:01:33.107929+0000","last_undegraded":"2024-03-28T15:01:33.107929+0000","last_fullsized":"2024-03-28T15:01:33.107929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460506+0000","last_change":"2024-03-28T15:01:32.113345+0000","last_active":"2024-03-28T15:01:33.460506+0000","last_peered":"2024-03-28T15:01:33.460506+0000","last_clean":"2024-03-28T15:01:33.460506+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:01:33.460506+0000","last_undegraded":"2024-03-28T15:01:33.460506+0000","last_fullsized":"2024-03-28T15:01:33.460506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107235+0000","last_change":"2024-03-28T15:01:32.112328+0000","last_active":"2024-03-28T15:01:33.107235+0000","last_peered":"2024-03-28T15:01:33.107235+0000","last_clean":"2024-03-28T15:01:33.107235+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:01:33.107235+0000","last_undegraded":"2024-03-28T15:01:33.107235+0000","last_fullsized":"2024-03-28T15:01:33.107235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461017+0000","last_change":"2024-03-28T15:01:32.122399+0000","last_active":"2024-03-28T15:01:33.461017+0000","last_peered":"2024-03-28T15:01:33.461017+0000","last_clean":"2024-03-28T15:01:33.461017+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:01:33.461017+0000","last_undegraded":"2024-03-28T15:01:33.461017+0000","last_fullsized":"2024-03-28T15:01:33.461017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460971+0000","last_change":"2024-03-28T15:01:32.123879+0000","last_active":"2024-03-28T15:01:33.460971+0000","last_peered":"2024-03-28T15:01:33.460971+0000","last_clean":"2024-03-28T15:01:33.460971+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:01:33.460971+0000","last_undegraded":"2024-03-28T15:01:33.460971+0000","last_fullsized":"2024-03-28T15:01:33.460971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104923+0000","last_change":"2024-03-28T15:01:32.110380+0000","last_active":"2024-03-28T15:01:33.104923+0000","last_peered":"2024-03-28T15:01:33.104923+0000","last_clean":"2024-03-28T15:01:33.104923+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:01:33.104923+0000","last_undegraded":"2024-03-28T15:01:33.104923+0000","last_fullsized":"2024-03-28T15:01:33.104923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460754+0000","last_change":"2024-03-28T15:01:32.113422+0000","last_active":"2024-03-28T15:01:33.460754+0000","last_peered":"2024-03-28T15:01:33.460754+0000","last_clean":"2024-03-28T15:01:33.460754+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:01:33.460754+0000","last_undegraded":"2024-03-28T15:01:33.460754+0000","last_fullsized":"2024-03-28T15:01:33.460754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.117560+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108157+0000","last_change":"2024-03-28T15:01:32.117440+0000","last_active":"2024-03-28T15:01:33.108157+0000","last_peered":"2024-03-28T15:01:33.108157+0000","last_clean":"2024-03-28T15:01:33.108157+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:01:33.108157+0000","last_undegraded":"2024-03-28T15:01:33.108157+0000","last_fullsized":"2024-03-28T15:01:33.108157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460388+0000","last_change":"2024-03-28T15:01:32.111344+0000","last_active":"2024-03-28T15:01:33.460388+0000","last_peered":"2024-03-28T15:01:33.460388+0000","last_clean":"2024-03-28T15:01:33.460388+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:01:33.460388+0000","last_undegraded":"2024-03-28T15:01:33.460388+0000","last_fullsized":"2024-03-28T15:01:33.460388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460295+0000","last_change":"2024-03-28T15:01:32.126811+0000","last_active":"2024-03-28T15:01:33.460295+0000","last_peered":"2024-03-28T15:01:33.460295+0000","last_clean":"2024-03-28T15:01:33.460295+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:01:33.460295+0000","last_undegraded":"2024-03-28T15:01:33.460295+0000","last_fullsized":"2024-03-28T15:01:33.460295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106843+0000","last_change":"2024-03-28T15:01:32.112050+0000","last_active":"2024-03-28T15:01:33.106843+0000","last_peered":"2024-03-28T15:01:33.106843+0000","last_clean":"2024-03-28T15:01:33.106843+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:01:33.106843+0000","last_undegraded":"2024-03-28T15:01:33.106843+0000","last_fullsized":"2024-03-28T15:01:33.106843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:30.376545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.475527+0000","last_change":"2024-03-28T15:01:31.097415+0000","last_active":"2024-03-28T15:01:34.475527+0000","last_peered":"2024-03-28T15:01:34.475527+0000","last_clean":"2024-03-28T15:01:34.475527+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:01:34.475527+0000","last_undegraded":"2024-03-28T15:01:34.475527+0000","last_fullsized":"2024-03-28T15:01:34.475527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461379+0000","last_change":"2024-03-28T15:01:32.123479+0000","last_active":"2024-03-28T15:01:33.461379+0000","last_peered":"2024-03-28T15:01:33.461379+0000","last_clean":"2024-03-28T15:01:33.461379+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:01:33.461379+0000","last_undegraded":"2024-03-28T15:01:33.461379+0000","last_fullsized":"2024-03-28T15:01:33.461379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.476969+0000","last_change":"2024-03-28T15:01:31.088437+0000","last_active":"2024-03-28T15:01:34.476969+0000","last_peered":"2024-03-28T15:01:34.476969+0000","last_clean":"2024-03-28T15:01:34.476969+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:01:34.476969+0000","last_undegraded":"2024-03-28T15:01:34.476969+0000","last_fullsized":"2024-03-28T15:01:34.476969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460403+0000","last_change":"2024-03-28T15:01:31.097499+0000","last_active":"2024-03-28T15:01:33.460403+0000","last_peered":"2024-03-28T15:01:33.460403+0000","last_clean":"2024-03-28T15:01:33.460403+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:01:33.460403+0000","last_undegraded":"2024-03-28T15:01:33.460403+0000","last_fullsized":"2024-03-28T15:01:33.460403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460343+0000","last_change":"2024-03-28T15:01:32.115727+0000","last_active":"2024-03-28T15:01:33.460343+0000","last_peered":"2024-03-28T15:01:33.460343+0000","last_clean":"2024-03-28T15:01:33.460343+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:01:33.460343+0000","last_undegraded":"2024-03-28T15:01:33.460343+0000","last_fullsized":"2024-03-28T15:01:33.460343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107881+0000","last_change":"2024-03-28T15:01:31.086252+0000","last_active":"2024-03-28T15:01:33.107881+0000","last_peered":"2024-03-28T15:01:33.107881+0000","last_clean":"2024-03-28T15:01:33.107881+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:01:33.107881+0000","last_undegraded":"2024-03-28T15:01:33.107881+0000","last_fullsized":"2024-03-28T15:01:33.107881+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104868+0000","last_change":"2024-03-28T15:01:32.110465+0000","last_active":"2024-03-28T15:01:33.104868+0000","last_peered":"2024-03-28T15:01:33.104868+0000","last_clean":"2024-03-28T15:01:33.104868+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:01:33.104868+0000","last_undegraded":"2024-03-28T15:01:33.104868+0000","last_fullsized":"2024-03-28T15:01:33.104868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460446+0000","last_change":"2024-03-28T15:01:31.092338+0000","last_active":"2024-03-28T15:01:33.460446+0000","last_peered":"2024-03-28T15:01:33.460446+0000","last_clean":"2024-03-28T15:01:33.460446+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:01:33.460446+0000","last_undegraded":"2024-03-28T15:01:33.460446+0000","last_fullsized":"2024-03-28T15:01:33.460446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107369+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.107369+0000","last_peered":"2024-03-28T15:01:33.107369+0000","last_clean":"2024-03-28T15:01:33.107369+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:01:33.107369+0000","last_undegraded":"2024-03-28T15:01:33.107369+0000","last_fullsized":"2024-03-28T15:01:33.107369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107482+0000","last_change":"2024-03-28T15:01:31.088909+0000","last_active":"2024-03-28T15:01:33.107482+0000","last_peered":"2024-03-28T15:01:33.107482+0000","last_clean":"2024-03-28T15:01:33.107482+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:01:33.107482+0000","last_undegraded":"2024-03-28T15:01:33.107482+0000","last_fullsized":"2024-03-28T15:01:33.107482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461141+0000","last_change":"2024-03-28T15:01:32.117188+0000","last_active":"2024-03-28T15:01:33.461141+0000","last_peered":"2024-03-28T15:01:33.461141+0000","last_clean":"2024-03-28T15:01:33.461141+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:01:33.461141+0000","last_undegraded":"2024-03-28T15:01:33.461141+0000","last_fullsized":"2024-03-28T15:01:33.461141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.121351+0000","last_change":"2024-03-28T15:01:31.095935+0000","last_active":"2024-03-28T15:01:37.121351+0000","last_peered":"2024-03-28T15:01:37.121351+0000","last_clean":"2024-03-28T15:01:37.121351+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:01:37.121351+0000","last_undegraded":"2024-03-28T15:01:37.121351+0000","last_fullsized":"2024-03-28T15:01:37.121351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107512+0000","last_change":"2024-03-28T15:01:32.120677+0000","last_active":"2024-03-28T15:01:33.107512+0000","last_peered":"2024-03-28T15:01:33.107512+0000","last_clean":"2024-03-28T15:01:33.107512+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.107512+0000","last_undegraded":"2024-03-28T15:01:33.107512+0000","last_fullsized":"2024-03-28T15:01:33.107512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:47.705969+0000","last_change":"2024-03-28T15:01:31.089380+0000","last_active":"2024-03-28T15:01:47.705969+0000","last_peered":"2024-03-28T15:01:47.705969+0000","last_clean":"2024-03-28T15:01:47.705969+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:01:47.705969+0000","last_undegraded":"2024-03-28T15:01:47.705969+0000","last_fullsized":"2024-03-28T15:01:47.705969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.110749+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.110332+0000","last_became_peered":"2024-03-28T15:01:32.110332+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.122627+0000","last_change":"2024-03-28T15:01:31.098172+0000","last_active":"2024-03-28T15:01:37.122627+0000","last_peered":"2024-03-28T15:01:37.122627+0000","last_clean":"2024-03-28T15:01:37.122627+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:01:37.122627+0000","last_undegraded":"2024-03-28T15:01:37.122627+0000","last_fullsized":"2024-03-28T15:01:37.122627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:16.015528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461045+0000","last_change":"2024-03-28T15:01:32.117690+0000","last_active":"2024-03-28T15:01:33.461045+0000","last_peered":"2024-03-28T15:01:33.461045+0000","last_clean":"2024-03-28T15:01:33.461045+0000","last_became_active":"2024-03-28T15:01:32.117412+0000","last_became_peered":"2024-03-28T15:01:32.117412+0000","last_unstale":"2024-03-28T15:01:33.461045+0000","last_undegraded":"2024-03-28T15:01:33.461045+0000","last_fullsized":"2024-03-28T15:01:33.461045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108124+0000","last_change":"2024-03-28T15:01:31.090519+0000","last_active":"2024-03-28T15:01:33.108124+0000","last_peered":"2024-03-28T15:01:33.108124+0000","last_clean":"2024-03-28T15:01:33.108124+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:01:33.108124+0000","last_undegraded":"2024-03-28T15:01:33.108124+0000","last_fullsized":"2024-03-28T15:01:33.108124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460932+0000","last_change":"2024-03-28T15:01:32.119827+0000","last_active":"2024-03-28T15:01:33.460932+0000","last_peered":"2024-03-28T15:01:33.460932+0000","last_clean":"2024-03-28T15:01:33.460932+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:01:33.460932+0000","last_undegraded":"2024-03-28T15:01:33.460932+0000","last_fullsized":"2024-03-28T15:01:33.460932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461234+0000","last_change":"2024-03-28T15:01:31.088364+0000","last_active":"2024-03-28T15:01:33.461234+0000","last_peered":"2024-03-28T15:01:33.461234+0000","last_clean":"2024-03-28T15:01:33.461234+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:01:33.461234+0000","last_undegraded":"2024-03-28T15:01:33.461234+0000","last_fullsized":"2024-03-28T15:01:33.461234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460422+0000","last_change":"2024-03-28T15:01:32.126177+0000","last_active":"2024-03-28T15:01:33.460422+0000","last_peered":"2024-03-28T15:01:33.460422+0000","last_clean":"2024-03-28T15:01:33.460422+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:01:33.460422+0000","last_undegraded":"2024-03-28T15:01:33.460422+0000","last_fullsized":"2024-03-28T15:01:33.460422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126124+0000","last_change":"2024-03-28T15:01:31.094837+0000","last_active":"2024-03-28T15:01:37.126124+0000","last_peered":"2024-03-28T15:01:37.126124+0000","last_clean":"2024-03-28T15:01:37.126124+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:01:37.126124+0000","last_undegraded":"2024-03-28T15:01:37.126124+0000","last_fullsized":"2024-03-28T15:01:37.126124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460675+0000","last_change":"2024-03-28T15:01:32.113803+0000","last_active":"2024-03-28T15:01:33.460675+0000","last_peered":"2024-03-28T15:01:33.460675+0000","last_clean":"2024-03-28T15:01:33.460675+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:01:33.460675+0000","last_undegraded":"2024-03-28T15:01:33.460675+0000","last_fullsized":"2024-03-28T15:01:33.460675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.477351+0000","last_change":"2024-03-28T15:01:31.092455+0000","last_active":"2024-03-28T15:01:34.477351+0000","last_peered":"2024-03-28T15:01:34.477351+0000","last_clean":"2024-03-28T15:01:34.477351+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:01:34.477351+0000","last_undegraded":"2024-03-28T15:01:34.477351+0000","last_fullsized":"2024-03-28T15:01:34.477351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:04.636216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460748+0000","last_change":"2024-03-28T15:01:32.117065+0000","last_active":"2024-03-28T15:01:33.460748+0000","last_peered":"2024-03-28T15:01:33.460748+0000","last_clean":"2024-03-28T15:01:33.460748+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:01:33.460748+0000","last_undegraded":"2024-03-28T15:01:33.460748+0000","last_fullsized":"2024-03-28T15:01:33.460748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.123413+0000","last_change":"2024-03-28T15:01:31.092175+0000","last_active":"2024-03-28T15:01:37.123413+0000","last_peered":"2024-03-28T15:01:37.123413+0000","last_clean":"2024-03-28T15:01:37.123413+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:01:37.123413+0000","last_undegraded":"2024-03-28T15:01:37.123413+0000","last_fullsized":"2024-03-28T15:01:37.123413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461324+0000","last_change":"2024-03-28T15:01:32.114975+0000","last_active":"2024-03-28T15:01:33.461324+0000","last_peered":"2024-03-28T15:01:33.461324+0000","last_clean":"2024-03-28T15:01:33.461324+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:01:33.461324+0000","last_undegraded":"2024-03-28T15:01:33.461324+0000","last_fullsized":"2024-03-28T15:01:33.461324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.482725+0000","last_change":"2024-03-28T15:01:31.097901+0000","last_active":"2024-03-28T15:01:34.482725+0000","last_peered":"2024-03-28T15:01:34.482725+0000","last_clean":"2024-03-28T15:01:34.482725+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:01:34.482725+0000","last_undegraded":"2024-03-28T15:01:34.482725+0000","last_fullsized":"2024-03-28T15:01:34.482725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108151+0000","last_change":"2024-03-28T15:01:32.120646+0000","last_active":"2024-03-28T15:01:33.108151+0000","last_peered":"2024-03-28T15:01:33.108151+0000","last_clean":"2024-03-28T15:01:33.108151+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:01:33.108151+0000","last_undegraded":"2024-03-28T15:01:33.108151+0000","last_fullsized":"2024-03-28T15:01:33.108151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460197+0000","last_change":"2024-03-28T15:01:31.086634+0000","last_active":"2024-03-28T15:01:33.460197+0000","last_peered":"2024-03-28T15:01:33.460197+0000","last_clean":"2024-03-28T15:01:33.460197+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:01:33.460197+0000","last_undegraded":"2024-03-28T15:01:33.460197+0000","last_fullsized":"2024-03-28T15:01:33.460197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461104+0000","last_change":"2024-03-28T15:01:32.112493+0000","last_active":"2024-03-28T15:01:33.461104+0000","last_peered":"2024-03-28T15:01:33.461104+0000","last_clean":"2024-03-28T15:01:33.461104+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:01:33.461104+0000","last_undegraded":"2024-03-28T15:01:33.461104+0000","last_fullsized":"2024-03-28T15:01:33.461104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461052+0000","last_change":"2024-03-28T15:01:20.724163+0000","last_active":"2024-03-28T15:01:33.461052+0000","last_peered":"2024-03-28T15:01:33.461052+0000","last_clean":"2024-03-28T15:01:33.461052+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:01:33.461052+0000","last_undegraded":"2024-03-28T15:01:33.461052+0000","last_fullsized":"2024-03-28T15:01:33.461052+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_clean_scrub_stamp":"2024-03-28T15:01:19.548090+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:15.791893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107058+0000","last_change":"2024-03-28T15:01:31.085453+0000","last_active":"2024-03-28T15:01:33.107058+0000","last_peered":"2024-03-28T15:01:33.107058+0000","last_clean":"2024-03-28T15:01:33.107058+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:01:33.107058+0000","last_undegraded":"2024-03-28T15:01:33.107058+0000","last_fullsized":"2024-03-28T15:01:33.107058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460326+0000","last_change":"2024-03-28T15:01:32.118678+0000","last_active":"2024-03-28T15:01:33.460326+0000","last_peered":"2024-03-28T15:01:33.460326+0000","last_clean":"2024-03-28T15:01:33.460326+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:01:33.460326+0000","last_undegraded":"2024-03-28T15:01:33.460326+0000","last_fullsized":"2024-03-28T15:01:33.460326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461116+0000","last_change":"2024-03-28T15:01:31.095439+0000","last_active":"2024-03-28T15:01:33.461116+0000","last_peered":"2024-03-28T15:01:33.461116+0000","last_clean":"2024-03-28T15:01:33.461116+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:01:33.461116+0000","last_undegraded":"2024-03-28T15:01:33.461116+0000","last_fullsized":"2024-03-28T15:01:33.461116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460849+0000","last_change":"2024-03-28T15:01:32.122655+0000","last_active":"2024-03-28T15:01:33.460849+0000","last_peered":"2024-03-28T15:01:33.460849+0000","last_clean":"2024-03-28T15:01:33.460849+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:01:33.460849+0000","last_undegraded":"2024-03-28T15:01:33.460849+0000","last_fullsized":"2024-03-28T15:01:33.460849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:05.942532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460385+0000","last_change":"2024-03-28T15:01:31.093265+0000","last_active":"2024-03-28T15:01:33.460385+0000","last_peered":"2024-03-28T15:01:33.460385+0000","last_clean":"2024-03-28T15:01:33.460385+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:01:33.460385+0000","last_undegraded":"2024-03-28T15:01:33.460385+0000","last_fullsized":"2024-03-28T15:01:33.460385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104892+0000","last_change":"2024-03-28T15:01:32.108597+0000","last_active":"2024-03-28T15:01:33.104892+0000","last_peered":"2024-03-28T15:01:33.104892+0000","last_clean":"2024-03-28T15:01:33.104892+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:01:33.104892+0000","last_undegraded":"2024-03-28T15:01:33.104892+0000","last_fullsized":"2024-03-28T15:01:33.104892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108209+0000","last_change":"2024-03-28T15:01:31.094750+0000","last_active":"2024-03-28T15:01:33.108209+0000","last_peered":"2024-03-28T15:01:33.108209+0000","last_clean":"2024-03-28T15:01:33.108209+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:01:33.108209+0000","last_undegraded":"2024-03-28T15:01:33.108209+0000","last_fullsized":"2024-03-28T15:01:33.108209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106970+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.106970+0000","last_peered":"2024-03-28T15:01:33.106970+0000","last_clean":"2024-03-28T15:01:33.106970+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:01:33.106970+0000","last_undegraded":"2024-03-28T15:01:33.106970+0000","last_fullsized":"2024-03-28T15:01:33.106970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:24.747778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104874+0000","last_change":"2024-03-28T15:01:31.093706+0000","last_active":"2024-03-28T15:01:33.104874+0000","last_peered":"2024-03-28T15:01:33.104874+0000","last_clean":"2024-03-28T15:01:33.104874+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:01:33.104874+0000","last_undegraded":"2024-03-28T15:01:33.104874+0000","last_fullsized":"2024-03-28T15:01:33.104874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108112+0000","last_change":"2024-03-28T15:01:32.122840+0000","last_active":"2024-03-28T15:01:33.108112+0000","last_peered":"2024-03-28T15:01:33.108112+0000","last_clean":"2024-03-28T15:01:33.108112+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:01:33.108112+0000","last_undegraded":"2024-03-28T15:01:33.108112+0000","last_fullsized":"2024-03-28T15:01:33.108112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461371+0000","last_change":"2024-03-28T15:01:31.095126+0000","last_active":"2024-03-28T15:01:33.461371+0000","last_peered":"2024-03-28T15:01:33.461371+0000","last_clean":"2024-03-28T15:01:33.461371+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:01:33.461371+0000","last_undegraded":"2024-03-28T15:01:33.461371+0000","last_fullsized":"2024-03-28T15:01:33.461371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:21.669535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104885+0000","last_change":"2024-03-28T15:01:32.111925+0000","last_active":"2024-03-28T15:01:33.104885+0000","last_peered":"2024-03-28T15:01:33.104885+0000","last_clean":"2024-03-28T15:01:33.104885+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:01:33.104885+0000","last_undegraded":"2024-03-28T15:01:33.104885+0000","last_fullsized":"2024-03-28T15:01:33.104885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.482739+0000","last_change":"2024-03-28T15:01:31.093197+0000","last_active":"2024-03-28T15:01:34.482739+0000","last_peered":"2024-03-28T15:01:34.482739+0000","last_clean":"2024-03-28T15:01:34.482739+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:01:34.482739+0000","last_undegraded":"2024-03-28T15:01:34.482739+0000","last_fullsized":"2024-03-28T15:01:34.482739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461153+0000","last_change":"2024-03-28T15:01:32.115432+0000","last_active":"2024-03-28T15:01:33.461153+0000","last_peered":"2024-03-28T15:01:33.461153+0000","last_clean":"2024-03-28T15:01:33.461153+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:01:33.461153+0000","last_undegraded":"2024-03-28T15:01:33.461153+0000","last_fullsized":"2024-03-28T15:01:33.461153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.125948+0000","last_change":"2024-03-28T15:01:31.088204+0000","last_active":"2024-03-28T15:01:37.125948+0000","last_peered":"2024-03-28T15:01:37.125948+0000","last_clean":"2024-03-28T15:01:37.125948+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:01:37.125948+0000","last_undegraded":"2024-03-28T15:01:37.125948+0000","last_fullsized":"2024-03-28T15:01:37.125948+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461225+0000","last_change":"2024-03-28T15:01:32.114331+0000","last_active":"2024-03-28T15:01:33.461225+0000","last_peered":"2024-03-28T15:01:33.461225+0000","last_clean":"2024-03-28T15:01:33.461225+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:01:33.461225+0000","last_undegraded":"2024-03-28T15:01:33.461225+0000","last_fullsized":"2024-03-28T15:01:33.461225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460969+0000","last_change":"2024-03-28T15:01:31.095371+0000","last_active":"2024-03-28T15:01:33.460969+0000","last_peered":"2024-03-28T15:01:33.460969+0000","last_clean":"2024-03-28T15:01:33.460969+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:01:33.460969+0000","last_undegraded":"2024-03-28T15:01:33.460969+0000","last_fullsized":"2024-03-28T15:01:33.460969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107409+0000","last_change":"2024-03-28T15:01:32.110950+0000","last_active":"2024-03-28T15:01:33.107409+0000","last_peered":"2024-03-28T15:01:33.107409+0000","last_clean":"2024-03-28T15:01:33.107409+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:01:33.107409+0000","last_undegraded":"2024-03-28T15:01:33.107409+0000","last_fullsized":"2024-03-28T15:01:33.107409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105009+0000","last_change":"2024-03-28T15:01:31.094542+0000","last_active":"2024-03-28T15:01:33.105009+0000","last_peered":"2024-03-28T15:01:33.105009+0000","last_clean":"2024-03-28T15:01:33.105009+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:01:33.105009+0000","last_undegraded":"2024-03-28T15:01:33.105009+0000","last_fullsized":"2024-03-28T15:01:33.105009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460466+0000","last_change":"2024-03-28T15:01:32.112158+0000","last_active":"2024-03-28T15:01:33.460466+0000","last_peered":"2024-03-28T15:01:33.460466+0000","last_clean":"2024-03-28T15:01:33.460466+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:01:33.460466+0000","last_undegraded":"2024-03-28T15:01:33.460466+0000","last_fullsized":"2024-03-28T15:01:33.460466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104884+0000","last_change":"2024-03-28T15:01:31.094440+0000","last_active":"2024-03-28T15:01:33.104884+0000","last_peered":"2024-03-28T15:01:33.104884+0000","last_clean":"2024-03-28T15:01:33.104884+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:01:33.104884+0000","last_undegraded":"2024-03-28T15:01:33.104884+0000","last_fullsized":"2024-03-28T15:01:33.104884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460758+0000","last_change":"2024-03-28T15:01:32.115192+0000","last_active":"2024-03-28T15:01:33.460758+0000","last_peered":"2024-03-28T15:01:33.460758+0000","last_clean":"2024-03-28T15:01:33.460758+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:01:33.460758+0000","last_undegraded":"2024-03-28T15:01:33.460758+0000","last_fullsized":"2024-03-28T15:01:33.460758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.125967+0000","last_change":"2024-03-28T15:01:31.096352+0000","last_active":"2024-03-28T15:01:37.125967+0000","last_peered":"2024-03-28T15:01:37.125967+0000","last_clean":"2024-03-28T15:01:37.125967+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:01:37.125967+0000","last_undegraded":"2024-03-28T15:01:37.125967+0000","last_fullsized":"2024-03-28T15:01:37.125967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460555+0000","last_change":"2024-03-28T15:01:32.124700+0000","last_active":"2024-03-28T15:01:33.460555+0000","last_peered":"2024-03-28T15:01:33.460555+0000","last_clean":"2024-03-28T15:01:33.460555+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:01:33.460555+0000","last_undegraded":"2024-03-28T15:01:33.460555+0000","last_fullsized":"2024-03-28T15:01:33.460555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107999+0000","last_change":"2024-03-28T15:01:32.123638+0000","last_active":"2024-03-28T15:01:33.107999+0000","last_peered":"2024-03-28T15:01:33.107999+0000","last_clean":"2024-03-28T15:01:33.107999+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:01:33.107999+0000","last_undegraded":"2024-03-28T15:01:33.107999+0000","last_fullsized":"2024-03-28T15:01:33.107999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.121539+0000","last_change":"2024-03-28T15:01:31.094281+0000","last_active":"2024-03-28T15:01:37.121539+0000","last_peered":"2024-03-28T15:01:37.121539+0000","last_clean":"2024-03-28T15:01:37.121539+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:01:37.121539+0000","last_undegraded":"2024-03-28T15:01:37.121539+0000","last_fullsized":"2024-03-28T15:01:37.121539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107988+0000","last_change":"2024-03-28T15:01:32.112496+0000","last_active":"2024-03-28T15:01:33.107988+0000","last_peered":"2024-03-28T15:01:33.107988+0000","last_clean":"2024-03-28T15:01:33.107988+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:01:33.107988+0000","last_undegraded":"2024-03-28T15:01:33.107988+0000","last_fullsized":"2024-03-28T15:01:33.107988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126177+0000","last_change":"2024-03-28T15:01:31.093773+0000","last_active":"2024-03-28T15:01:37.126177+0000","last_peered":"2024-03-28T15:01:37.126177+0000","last_clean":"2024-03-28T15:01:37.126177+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:01:37.126177+0000","last_undegraded":"2024-03-28T15:01:37.126177+0000","last_fullsized":"2024-03-28T15:01:37.126177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:54.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460976+0000","last_change":"2024-03-28T15:01:32.117275+0000","last_active":"2024-03-28T15:01:33.460976+0000","last_peered":"2024-03-28T15:01:33.460976+0000","last_clean":"2024-03-28T15:01:33.460976+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:01:33.460976+0000","last_undegraded":"2024-03-28T15:01:33.460976+0000","last_fullsized":"2024-03-28T15:01:33.460976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.475554+0000","last_change":"2024-03-28T15:01:31.096725+0000","last_active":"2024-03-28T15:01:34.475554+0000","last_peered":"2024-03-28T15:01:34.475554+0000","last_clean":"2024-03-28T15:01:34.475554+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:01:34.475554+0000","last_undegraded":"2024-03-28T15:01:34.475554+0000","last_fullsized":"2024-03-28T15:01:34.475554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460734+0000","last_change":"2024-03-28T15:01:32.120071+0000","last_active":"2024-03-28T15:01:33.460734+0000","last_peered":"2024-03-28T15:01:33.460734+0000","last_clean":"2024-03-28T15:01:33.460734+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:01:33.460734+0000","last_undegraded":"2024-03-28T15:01:33.460734+0000","last_fullsized":"2024-03-28T15:01:33.460734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126217+0000","last_change":"2024-03-28T15:01:31.095351+0000","last_active":"2024-03-28T15:01:37.126217+0000","last_peered":"2024-03-28T15:01:37.126217+0000","last_clean":"2024-03-28T15:01:37.126217+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:01:37.126217+0000","last_undegraded":"2024-03-28T15:01:37.126217+0000","last_fullsized":"2024-03-28T15:01:37.126217+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":73802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":73802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:01:51.218 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:01:51.354 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.354+0000 7f70ac777640 1 Processor -- start 2024-03-28T15:01:51.354 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.354+0000 7f70ac777640 1 -- start start 2024-03-28T15:01:51.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70ac777640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f70a40c8090 0x7f70a4158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70ac777640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a4072df0 con 0x7f70a4071340 2024-03-28T15:01:51.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70ac777640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a40c8660 con 0x7f70a4152810 2024-03-28T15:01:51.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70ac777640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f70a4159500 con 0x7f70a40c8090 2024-03-28T15:01:51.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70aad14640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f70a40c8090 0x7f70a4158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70a9d12640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f70a4071340 0x7f70a40c6cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49786/0 (socket says 172.21.15.186:49786) 2024-03-28T15:01:51.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70aad14640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f70a40c8090 0x7f70a4158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42380/0 (socket says 172.21.15.186:42380) 2024-03-28T15:01:51.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70a9d12640 1 -- 172.21.15.186:0/1981876320 learned_addr learned my addr 172.21.15.186:0/1981876320 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:51.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70aad14640 1 -- 172.21.15.186:0/1981876320 >> v1:172.21.15.191:6789/0 conn(0x7f70a4152810 legacy=0x7f70a406c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70aad14640 1 -- 172.21.15.186:0/1981876320 >> v1:172.21.15.186:6789/0 conn(0x7f70a4071340 legacy=0x7f70a40c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70a9511640 1 -- 172.21.15.186:0/1981876320 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 455591346 0 0) 0x7f70a40c8660 con 0x7f70a4152810 2024-03-28T15:01:51.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.355+0000 7f70aad14640 1 -- 172.21.15.186:0/1981876320 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a4071d50 con 0x7f70a40c8090 2024-03-28T15:01:51.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.356+0000 7f70aad14640 1 --2- 172.21.15.186:0/1981876320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70a40c8090 0x7f70a4158fc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f709c0097a0 tx=0x7f709c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e338719e10786ce1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:51.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.356+0000 7f70a9511640 1 -- 172.21.15.186:0/1981876320 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c037070 con 0x7f70a40c8090 2024-03-28T15:01:51.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.356+0000 7f70a9511640 1 -- 172.21.15.186:0/1981876320 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f709c030710 con 0x7f70a40c8090 2024-03-28T15:01:51.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.356+0000 7f70a9511640 1 -- 172.21.15.186:0/1981876320 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c030ae0 con 0x7f70a40c8090 2024-03-28T15:01:51.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.357+0000 7f70ac777640 1 -- 172.21.15.186:0/1981876320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70a40c8090 msgr2=0x7f70a4158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.357+0000 7f70ac777640 1 --2- 172.21.15.186:0/1981876320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70a40c8090 0x7f70a4158fc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f709c0097a0 tx=0x7f709c02b790 comp rx=0 tx=0).stop 2024-03-28T15:01:51.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.357+0000 7f70ac777640 1 -- 172.21.15.186:0/1981876320 shutdown_connections 2024-03-28T15:01:51.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.357+0000 7f70ac777640 1 --2- 172.21.15.186:0/1981876320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70a40c8090 0x7f70a4158fc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.357+0000 7f70ac777640 1 -- 172.21.15.186:0/1981876320 >> 172.21.15.186:0/1981876320 conn(0x7f70a40c8a20 msgr2=0x7f70a40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:51.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.357+0000 7f70ac777640 1 -- 172.21.15.186:0/1981876320 shutdown_connections 2024-03-28T15:01:51.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.357+0000 7f70ac777640 1 -- 172.21.15.186:0/1981876320 wait complete. 2024-03-28T15:01:51.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.358+0000 7f70ac777640 1 Processor -- start 2024-03-28T15:01:51.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.358+0000 7f70ac777640 1 -- start start 2024-03-28T15:01:51.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70ac777640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70a4071340 0x7f70a40c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70ac777640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70a40c8090 0x7f70a41563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70ac777640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70a4152810 0x7f70a406f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70ac777640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f70a40725f0 con 0x7f70a4071340 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70ac777640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4072310 con 0x7f70a40c8090 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70ac777640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f70a40728d0 con 0x7f70a4152810 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70aa513640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70a4071340 0x7f70a40c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70aa513640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70a4071340 0x7f70a40c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42394/0 (socket says 172.21.15.186:42394) 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70aa513640 1 -- 172.21.15.186:0/123460579 learned_addr learned my addr 172.21.15.186:0/123460579 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70aad14640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70a4152810 0x7f70a406f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70a9d12640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70a40c8090 0x7f70a41563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70aa513640 1 -- 172.21.15.186:0/123460579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70a4152810 msgr2=0x7f70a406f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70aa513640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70a4152810 0x7f70a406f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70aa513640 1 -- 172.21.15.186:0/123460579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70a40c8090 msgr2=0x7f70a41563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70aa513640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70a40c8090 0x7f70a41563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70aa513640 1 -- 172.21.15.186:0/123460579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a40c1a10 con 0x7f70a4071340 2024-03-28T15:01:51.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70aa513640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70a4071340 0x7f70a40c21b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f70a000c890 tx=0x7f70a000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:51.362 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f709affd640 1 -- 172.21.15.186:0/123460579 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0017070 con 0x7f70a4071340 2024-03-28T15:01:51.362 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f709affd640 1 -- 172.21.15.186:0/123460579 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a0003dc0 con 0x7f70a4071340 2024-03-28T15:01:51.362 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f709affd640 1 -- 172.21.15.186:0/123460579 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a00052b0 con 0x7f70a4071340 2024-03-28T15:01:51.362 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.359+0000 7f70ac777640 1 -- 172.21.15.186:0/123460579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a4157d40 con 0x7f70a4071340 2024-03-28T15:01:51.362 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.360+0000 7f70ac777640 1 -- 172.21.15.186:0/123460579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a40b13d0 con 0x7f70a4071340 2024-03-28T15:01:51.362 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.360+0000 7f70ac777640 1 -- 172.21.15.186:0/123460579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a40c57b0 con 0x7f70a4071340 2024-03-28T15:01:51.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.361+0000 7f709affd640 1 -- 172.21.15.186:0/123460579 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f70a001a070 con 0x7f70a4071340 2024-03-28T15:01:51.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.361+0000 7f709affd640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f708806c530 0x7f708806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.361+0000 7f709affd640 1 -- 172.21.15.186:0/123460579 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f70a008ae00 con 0x7f70a4071340 2024-03-28T15:01:51.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.361+0000 7f70a9d12640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f708806c530 0x7f708806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.362+0000 7f70a9d12640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f708806c530 0x7f708806e9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f70a41e1510 tx=0x7f7094007e00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:51.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.366+0000 7f709affd640 1 -- 172.21.15.186:0/123460579 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f70a40c57b0 con 0x7f70a4071340 2024-03-28T15:01:51.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.540+0000 7f70ac777640 1 -- 172.21.15.186:0/123460579 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f70a40c1850 con 0x7f708806c530 2024-03-28T15:01:51.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.546+0000 7f709affd640 1 -- 172.21.15.186:0/123460579 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251831 (secure 0 0 0) 0x7f7094004c70 con 0x7f708806c530 2024-03-28T15:01:51.549 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:51.550 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:01:51.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 -- 172.21.15.186:0/123460579 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f708806c530 msgr2=0x7f708806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f708806c530 0x7f708806e9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f70a41e1510 tx=0x7f7094007e00 comp rx=0 tx=0).stop 2024-03-28T15:01:51.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 -- 172.21.15.186:0/123460579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70a4071340 msgr2=0x7f70a40c21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70a4071340 0x7f70a40c21b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f70a000c890 tx=0x7f70a000cd50 comp rx=0 tx=0).stop 2024-03-28T15:01:51.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 -- 172.21.15.186:0/123460579 shutdown_connections 2024-03-28T15:01:51.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f708806c530 0x7f708806e9f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70a4152810 0x7f70a406f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70a40c8090 0x7f70a41563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 --2- 172.21.15.186:0/123460579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70a4071340 0x7f70a40c21b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 -- 172.21.15.186:0/123460579 >> 172.21.15.186:0/123460579 conn(0x7f70a40c8a20 msgr2=0x7f70a406c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:51.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 -- 172.21.15.186:0/123460579 shutdown_connections 2024-03-28T15:01:51.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.549+0000 7f70ac777640 1 -- 172.21.15.186:0/123460579 wait complete. 2024-03-28T15:01:51.560 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-28T15:01:50.638678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594034,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218952,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755768,"statfs":{"total":773094113280,"available":772869906432,"internally_reserved":0,"allocated":2891776,"data_stored":1755540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1256,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002386"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461304+0000","last_change":"2024-03-28T15:01:32.122226+0000","last_active":"2024-03-28T15:01:33.461304+0000","last_peered":"2024-03-28T15:01:33.461304+0000","last_clean":"2024-03-28T15:01:33.461304+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:01:33.461304+0000","last_undegraded":"2024-03-28T15:01:33.461304+0000","last_fullsized":"2024-03-28T15:01:33.461304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104984+0000","last_change":"2024-03-28T15:01:32.115850+0000","last_active":"2024-03-28T15:01:33.104984+0000","last_peered":"2024-03-28T15:01:33.104984+0000","last_clean":"2024-03-28T15:01:33.104984+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:01:33.104984+0000","last_undegraded":"2024-03-28T15:01:33.104984+0000","last_fullsized":"2024-03-28T15:01:33.104984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.119744+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:57.022614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107952+0000","last_change":"2024-03-28T15:01:32.120916+0000","last_active":"2024-03-28T15:01:33.107952+0000","last_peered":"2024-03-28T15:01:33.107952+0000","last_clean":"2024-03-28T15:01:33.107952+0000","last_became_active":"2024-03-28T15:01:32.117449+0000","last_became_peered":"2024-03-28T15:01:32.117449+0000","last_unstale":"2024-03-28T15:01:33.107952+0000","last_undegraded":"2024-03-28T15:01:33.107952+0000","last_fullsized":"2024-03-28T15:01:33.107952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461016+0000","last_change":"2024-03-28T15:01:32.112835+0000","last_active":"2024-03-28T15:01:33.461016+0000","last_peered":"2024-03-28T15:01:33.461016+0000","last_clean":"2024-03-28T15:01:33.461016+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:01:33.461016+0000","last_undegraded":"2024-03-28T15:01:33.461016+0000","last_fullsized":"2024-03-28T15:01:33.461016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107714+0000","last_change":"2024-03-28T15:01:32.116248+0000","last_active":"2024-03-28T15:01:33.107714+0000","last_peered":"2024-03-28T15:01:33.107714+0000","last_clean":"2024-03-28T15:01:33.107714+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:01:33.107714+0000","last_undegraded":"2024-03-28T15:01:33.107714+0000","last_fullsized":"2024-03-28T15:01:33.107714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460486+0000","last_change":"2024-03-28T15:01:32.115221+0000","last_active":"2024-03-28T15:01:33.460486+0000","last_peered":"2024-03-28T15:01:33.460486+0000","last_clean":"2024-03-28T15:01:33.460486+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:01:33.460486+0000","last_undegraded":"2024-03-28T15:01:33.460486+0000","last_fullsized":"2024-03-28T15:01:33.460486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460218+0000","last_change":"2024-03-28T15:01:32.118310+0000","last_active":"2024-03-28T15:01:33.460218+0000","last_peered":"2024-03-28T15:01:33.460218+0000","last_clean":"2024-03-28T15:01:33.460218+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:01:33.460218+0000","last_undegraded":"2024-03-28T15:01:33.460218+0000","last_fullsized":"2024-03-28T15:01:33.460218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461212+0000","last_change":"2024-03-28T15:01:32.114205+0000","last_active":"2024-03-28T15:01:33.461212+0000","last_peered":"2024-03-28T15:01:33.461212+0000","last_clean":"2024-03-28T15:01:33.461212+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:01:33.461212+0000","last_undegraded":"2024-03-28T15:01:33.461212+0000","last_fullsized":"2024-03-28T15:01:33.461212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460905+0000","last_change":"2024-03-28T15:01:32.125178+0000","last_active":"2024-03-28T15:01:33.460905+0000","last_peered":"2024-03-28T15:01:33.460905+0000","last_clean":"2024-03-28T15:01:33.460905+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:01:33.460905+0000","last_undegraded":"2024-03-28T15:01:33.460905+0000","last_fullsized":"2024-03-28T15:01:33.460905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460676+0000","last_change":"2024-03-28T15:01:32.125839+0000","last_active":"2024-03-28T15:01:33.460676+0000","last_peered":"2024-03-28T15:01:33.460676+0000","last_clean":"2024-03-28T15:01:33.460676+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:01:33.460676+0000","last_undegraded":"2024-03-28T15:01:33.460676+0000","last_fullsized":"2024-03-28T15:01:33.460676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107231+0000","last_change":"2024-03-28T15:01:32.121874+0000","last_active":"2024-03-28T15:01:33.107231+0000","last_peered":"2024-03-28T15:01:33.107231+0000","last_clean":"2024-03-28T15:01:33.107231+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:01:33.107231+0000","last_undegraded":"2024-03-28T15:01:33.107231+0000","last_fullsized":"2024-03-28T15:01:33.107231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:20.744143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460735+0000","last_change":"2024-03-28T15:01:32.119348+0000","last_active":"2024-03-28T15:01:33.460735+0000","last_peered":"2024-03-28T15:01:33.460735+0000","last_clean":"2024-03-28T15:01:33.460735+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:01:33.460735+0000","last_undegraded":"2024-03-28T15:01:33.460735+0000","last_fullsized":"2024-03-28T15:01:33.460735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461204+0000","last_change":"2024-03-28T15:01:32.114290+0000","last_active":"2024-03-28T15:01:33.461204+0000","last_peered":"2024-03-28T15:01:33.461204+0000","last_clean":"2024-03-28T15:01:33.461204+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:01:33.461204+0000","last_undegraded":"2024-03-28T15:01:33.461204+0000","last_fullsized":"2024-03-28T15:01:33.461204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105071+0000","last_change":"2024-03-28T15:01:32.113654+0000","last_active":"2024-03-28T15:01:33.105071+0000","last_peered":"2024-03-28T15:01:33.105071+0000","last_clean":"2024-03-28T15:01:33.105071+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:01:33.105071+0000","last_undegraded":"2024-03-28T15:01:33.105071+0000","last_fullsized":"2024-03-28T15:01:33.105071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107620+0000","last_change":"2024-03-28T15:01:32.115901+0000","last_active":"2024-03-28T15:01:33.107620+0000","last_peered":"2024-03-28T15:01:33.107620+0000","last_clean":"2024-03-28T15:01:33.107620+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:01:33.107620+0000","last_undegraded":"2024-03-28T15:01:33.107620+0000","last_fullsized":"2024-03-28T15:01:33.107620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108153+0000","last_change":"2024-03-28T15:01:32.113156+0000","last_active":"2024-03-28T15:01:33.108153+0000","last_peered":"2024-03-28T15:01:33.108153+0000","last_clean":"2024-03-28T15:01:33.108153+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:01:33.108153+0000","last_undegraded":"2024-03-28T15:01:33.108153+0000","last_fullsized":"2024-03-28T15:01:33.108153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460557+0000","last_change":"2024-03-28T15:01:32.119785+0000","last_active":"2024-03-28T15:01:33.460557+0000","last_peered":"2024-03-28T15:01:33.460557+0000","last_clean":"2024-03-28T15:01:33.460557+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:01:33.460557+0000","last_undegraded":"2024-03-28T15:01:33.460557+0000","last_fullsized":"2024-03-28T15:01:33.460557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461426+0000","last_change":"2024-03-28T15:01:32.111912+0000","last_active":"2024-03-28T15:01:33.461426+0000","last_peered":"2024-03-28T15:01:33.461426+0000","last_clean":"2024-03-28T15:01:33.461426+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:01:33.461426+0000","last_undegraded":"2024-03-28T15:01:33.461426+0000","last_fullsized":"2024-03-28T15:01:33.461426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:52.632729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461025+0000","last_change":"2024-03-28T15:01:32.124381+0000","last_active":"2024-03-28T15:01:33.461025+0000","last_peered":"2024-03-28T15:01:33.461025+0000","last_clean":"2024-03-28T15:01:33.461025+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:01:33.461025+0000","last_undegraded":"2024-03-28T15:01:33.461025+0000","last_fullsized":"2024-03-28T15:01:33.461025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.120110+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107929+0000","last_change":"2024-03-28T15:01:32.113699+0000","last_active":"2024-03-28T15:01:33.107929+0000","last_peered":"2024-03-28T15:01:33.107929+0000","last_clean":"2024-03-28T15:01:33.107929+0000","last_became_active":"2024-03-28T15:01:32.113321+0000","last_became_peered":"2024-03-28T15:01:32.113321+0000","last_unstale":"2024-03-28T15:01:33.107929+0000","last_undegraded":"2024-03-28T15:01:33.107929+0000","last_fullsized":"2024-03-28T15:01:33.107929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460506+0000","last_change":"2024-03-28T15:01:32.113345+0000","last_active":"2024-03-28T15:01:33.460506+0000","last_peered":"2024-03-28T15:01:33.460506+0000","last_clean":"2024-03-28T15:01:33.460506+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:01:33.460506+0000","last_undegraded":"2024-03-28T15:01:33.460506+0000","last_fullsized":"2024-03-28T15:01:33.460506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107235+0000","last_change":"2024-03-28T15:01:32.112328+0000","last_active":"2024-03-28T15:01:33.107235+0000","last_peered":"2024-03-28T15:01:33.107235+0000","last_clean":"2024-03-28T15:01:33.107235+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:01:33.107235+0000","last_undegraded":"2024-03-28T15:01:33.107235+0000","last_fullsized":"2024-03-28T15:01:33.107235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461017+0000","last_change":"2024-03-28T15:01:32.122399+0000","last_active":"2024-03-28T15:01:33.461017+0000","last_peered":"2024-03-28T15:01:33.461017+0000","last_clean":"2024-03-28T15:01:33.461017+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:01:33.461017+0000","last_undegraded":"2024-03-28T15:01:33.461017+0000","last_fullsized":"2024-03-28T15:01:33.461017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460971+0000","last_change":"2024-03-28T15:01:32.123879+0000","last_active":"2024-03-28T15:01:33.460971+0000","last_peered":"2024-03-28T15:01:33.460971+0000","last_clean":"2024-03-28T15:01:33.460971+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:01:33.460971+0000","last_undegraded":"2024-03-28T15:01:33.460971+0000","last_fullsized":"2024-03-28T15:01:33.460971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104923+0000","last_change":"2024-03-28T15:01:32.110380+0000","last_active":"2024-03-28T15:01:33.104923+0000","last_peered":"2024-03-28T15:01:33.104923+0000","last_clean":"2024-03-28T15:01:33.104923+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:01:33.104923+0000","last_undegraded":"2024-03-28T15:01:33.104923+0000","last_fullsized":"2024-03-28T15:01:33.104923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460754+0000","last_change":"2024-03-28T15:01:32.113422+0000","last_active":"2024-03-28T15:01:33.460754+0000","last_peered":"2024-03-28T15:01:33.460754+0000","last_clean":"2024-03-28T15:01:33.460754+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:01:33.460754+0000","last_undegraded":"2024-03-28T15:01:33.460754+0000","last_fullsized":"2024-03-28T15:01:33.460754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.117560+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108157+0000","last_change":"2024-03-28T15:01:32.117440+0000","last_active":"2024-03-28T15:01:33.108157+0000","last_peered":"2024-03-28T15:01:33.108157+0000","last_clean":"2024-03-28T15:01:33.108157+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:01:33.108157+0000","last_undegraded":"2024-03-28T15:01:33.108157+0000","last_fullsized":"2024-03-28T15:01:33.108157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460388+0000","last_change":"2024-03-28T15:01:32.111344+0000","last_active":"2024-03-28T15:01:33.460388+0000","last_peered":"2024-03-28T15:01:33.460388+0000","last_clean":"2024-03-28T15:01:33.460388+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:01:33.460388+0000","last_undegraded":"2024-03-28T15:01:33.460388+0000","last_fullsized":"2024-03-28T15:01:33.460388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460295+0000","last_change":"2024-03-28T15:01:32.126811+0000","last_active":"2024-03-28T15:01:33.460295+0000","last_peered":"2024-03-28T15:01:33.460295+0000","last_clean":"2024-03-28T15:01:33.460295+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:01:33.460295+0000","last_undegraded":"2024-03-28T15:01:33.460295+0000","last_fullsized":"2024-03-28T15:01:33.460295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106843+0000","last_change":"2024-03-28T15:01:32.112050+0000","last_active":"2024-03-28T15:01:33.106843+0000","last_peered":"2024-03-28T15:01:33.106843+0000","last_clean":"2024-03-28T15:01:33.106843+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:01:33.106843+0000","last_undegraded":"2024-03-28T15:01:33.106843+0000","last_fullsized":"2024-03-28T15:01:33.106843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:30.376545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.475527+0000","last_change":"2024-03-28T15:01:31.097415+0000","last_active":"2024-03-28T15:01:34.475527+0000","last_peered":"2024-03-28T15:01:34.475527+0000","last_clean":"2024-03-28T15:01:34.475527+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:01:34.475527+0000","last_undegraded":"2024-03-28T15:01:34.475527+0000","last_fullsized":"2024-03-28T15:01:34.475527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461379+0000","last_change":"2024-03-28T15:01:32.123479+0000","last_active":"2024-03-28T15:01:33.461379+0000","last_peered":"2024-03-28T15:01:33.461379+0000","last_clean":"2024-03-28T15:01:33.461379+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:01:33.461379+0000","last_undegraded":"2024-03-28T15:01:33.461379+0000","last_fullsized":"2024-03-28T15:01:33.461379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.476969+0000","last_change":"2024-03-28T15:01:31.088437+0000","last_active":"2024-03-28T15:01:34.476969+0000","last_peered":"2024-03-28T15:01:34.476969+0000","last_clean":"2024-03-28T15:01:34.476969+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:01:34.476969+0000","last_undegraded":"2024-03-28T15:01:34.476969+0000","last_fullsized":"2024-03-28T15:01:34.476969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460403+0000","last_change":"2024-03-28T15:01:31.097499+0000","last_active":"2024-03-28T15:01:33.460403+0000","last_peered":"2024-03-28T15:01:33.460403+0000","last_clean":"2024-03-28T15:01:33.460403+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:01:33.460403+0000","last_undegraded":"2024-03-28T15:01:33.460403+0000","last_fullsized":"2024-03-28T15:01:33.460403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460343+0000","last_change":"2024-03-28T15:01:32.115727+0000","last_active":"2024-03-28T15:01:33.460343+0000","last_peered":"2024-03-28T15:01:33.460343+0000","last_clean":"2024-03-28T15:01:33.460343+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:01:33.460343+0000","last_undegraded":"2024-03-28T15:01:33.460343+0000","last_fullsized":"2024-03-28T15:01:33.460343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107881+0000","last_change":"2024-03-28T15:01:31.086252+0000","last_active":"2024-03-28T15:01:33.107881+0000","last_peered":"2024-03-28T15:01:33.107881+0000","last_clean":"2024-03-28T15:01:33.107881+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:01:33.107881+0000","last_undegraded":"2024-03-28T15:01:33.107881+0000","last_fullsized":"2024-03-28T15:01:33.107881+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104868+0000","last_change":"2024-03-28T15:01:32.110465+0000","last_active":"2024-03-28T15:01:33.104868+0000","last_peered":"2024-03-28T15:01:33.104868+0000","last_clean":"2024-03-28T15:01:33.104868+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:01:33.104868+0000","last_undegraded":"2024-03-28T15:01:33.104868+0000","last_fullsized":"2024-03-28T15:01:33.104868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460446+0000","last_change":"2024-03-28T15:01:31.092338+0000","last_active":"2024-03-28T15:01:33.460446+0000","last_peered":"2024-03-28T15:01:33.460446+0000","last_clean":"2024-03-28T15:01:33.460446+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:01:33.460446+0000","last_undegraded":"2024-03-28T15:01:33.460446+0000","last_fullsized":"2024-03-28T15:01:33.460446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107369+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.107369+0000","last_peered":"2024-03-28T15:01:33.107369+0000","last_clean":"2024-03-28T15:01:33.107369+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:01:33.107369+0000","last_undegraded":"2024-03-28T15:01:33.107369+0000","last_fullsized":"2024-03-28T15:01:33.107369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107482+0000","last_change":"2024-03-28T15:01:31.088909+0000","last_active":"2024-03-28T15:01:33.107482+0000","last_peered":"2024-03-28T15:01:33.107482+0000","last_clean":"2024-03-28T15:01:33.107482+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:01:33.107482+0000","last_undegraded":"2024-03-28T15:01:33.107482+0000","last_fullsized":"2024-03-28T15:01:33.107482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461141+0000","last_change":"2024-03-28T15:01:32.117188+0000","last_active":"2024-03-28T15:01:33.461141+0000","last_peered":"2024-03-28T15:01:33.461141+0000","last_clean":"2024-03-28T15:01:33.461141+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:01:33.461141+0000","last_undegraded":"2024-03-28T15:01:33.461141+0000","last_fullsized":"2024-03-28T15:01:33.461141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.121351+0000","last_change":"2024-03-28T15:01:31.095935+0000","last_active":"2024-03-28T15:01:37.121351+0000","last_peered":"2024-03-28T15:01:37.121351+0000","last_clean":"2024-03-28T15:01:37.121351+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:01:37.121351+0000","last_undegraded":"2024-03-28T15:01:37.121351+0000","last_fullsized":"2024-03-28T15:01:37.121351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107512+0000","last_change":"2024-03-28T15:01:32.120677+0000","last_active":"2024-03-28T15:01:33.107512+0000","last_peered":"2024-03-28T15:01:33.107512+0000","last_clean":"2024-03-28T15:01:33.107512+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.107512+0000","last_undegraded":"2024-03-28T15:01:33.107512+0000","last_fullsized":"2024-03-28T15:01:33.107512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:47.705969+0000","last_change":"2024-03-28T15:01:31.089380+0000","last_active":"2024-03-28T15:01:47.705969+0000","last_peered":"2024-03-28T15:01:47.705969+0000","last_clean":"2024-03-28T15:01:47.705969+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:01:47.705969+0000","last_undegraded":"2024-03-28T15:01:47.705969+0000","last_fullsized":"2024-03-28T15:01:47.705969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.110749+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.110332+0000","last_became_peered":"2024-03-28T15:01:32.110332+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.122627+0000","last_change":"2024-03-28T15:01:31.098172+0000","last_active":"2024-03-28T15:01:37.122627+0000","last_peered":"2024-03-28T15:01:37.122627+0000","last_clean":"2024-03-28T15:01:37.122627+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:01:37.122627+0000","last_undegraded":"2024-03-28T15:01:37.122627+0000","last_fullsized":"2024-03-28T15:01:37.122627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:16.015528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461045+0000","last_change":"2024-03-28T15:01:32.117690+0000","last_active":"2024-03-28T15:01:33.461045+0000","last_peered":"2024-03-28T15:01:33.461045+0000","last_clean":"2024-03-28T15:01:33.461045+0000","last_became_active":"2024-03-28T15:01:32.117412+0000","last_became_peered":"2024-03-28T15:01:32.117412+0000","last_unstale":"2024-03-28T15:01:33.461045+0000","last_undegraded":"2024-03-28T15:01:33.461045+0000","last_fullsized":"2024-03-28T15:01:33.461045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108124+0000","last_change":"2024-03-28T15:01:31.090519+0000","last_active":"2024-03-28T15:01:33.108124+0000","last_peered":"2024-03-28T15:01:33.108124+0000","last_clean":"2024-03-28T15:01:33.108124+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:01:33.108124+0000","last_undegraded":"2024-03-28T15:01:33.108124+0000","last_fullsized":"2024-03-28T15:01:33.108124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460932+0000","last_change":"2024-03-28T15:01:32.119827+0000","last_active":"2024-03-28T15:01:33.460932+0000","last_peered":"2024-03-28T15:01:33.460932+0000","last_clean":"2024-03-28T15:01:33.460932+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:01:33.460932+0000","last_undegraded":"2024-03-28T15:01:33.460932+0000","last_fullsized":"2024-03-28T15:01:33.460932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461234+0000","last_change":"2024-03-28T15:01:31.088364+0000","last_active":"2024-03-28T15:01:33.461234+0000","last_peered":"2024-03-28T15:01:33.461234+0000","last_clean":"2024-03-28T15:01:33.461234+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:01:33.461234+0000","last_undegraded":"2024-03-28T15:01:33.461234+0000","last_fullsized":"2024-03-28T15:01:33.461234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460422+0000","last_change":"2024-03-28T15:01:32.126177+0000","last_active":"2024-03-28T15:01:33.460422+0000","last_peered":"2024-03-28T15:01:33.460422+0000","last_clean":"2024-03-28T15:01:33.460422+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:01:33.460422+0000","last_undegraded":"2024-03-28T15:01:33.460422+0000","last_fullsized":"2024-03-28T15:01:33.460422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126124+0000","last_change":"2024-03-28T15:01:31.094837+0000","last_active":"2024-03-28T15:01:37.126124+0000","last_peered":"2024-03-28T15:01:37.126124+0000","last_clean":"2024-03-28T15:01:37.126124+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:01:37.126124+0000","last_undegraded":"2024-03-28T15:01:37.126124+0000","last_fullsized":"2024-03-28T15:01:37.126124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460675+0000","last_change":"2024-03-28T15:01:32.113803+0000","last_active":"2024-03-28T15:01:33.460675+0000","last_peered":"2024-03-28T15:01:33.460675+0000","last_clean":"2024-03-28T15:01:33.460675+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:01:33.460675+0000","last_undegraded":"2024-03-28T15:01:33.460675+0000","last_fullsized":"2024-03-28T15:01:33.460675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.477351+0000","last_change":"2024-03-28T15:01:31.092455+0000","last_active":"2024-03-28T15:01:34.477351+0000","last_peered":"2024-03-28T15:01:34.477351+0000","last_clean":"2024-03-28T15:01:34.477351+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:01:34.477351+0000","last_undegraded":"2024-03-28T15:01:34.477351+0000","last_fullsized":"2024-03-28T15:01:34.477351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:04.636216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460748+0000","last_change":"2024-03-28T15:01:32.117065+0000","last_active":"2024-03-28T15:01:33.460748+0000","last_peered":"2024-03-28T15:01:33.460748+0000","last_clean":"2024-03-28T15:01:33.460748+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:01:33.460748+0000","last_undegraded":"2024-03-28T15:01:33.460748+0000","last_fullsized":"2024-03-28T15:01:33.460748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.123413+0000","last_change":"2024-03-28T15:01:31.092175+0000","last_active":"2024-03-28T15:01:37.123413+0000","last_peered":"2024-03-28T15:01:37.123413+0000","last_clean":"2024-03-28T15:01:37.123413+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:01:37.123413+0000","last_undegraded":"2024-03-28T15:01:37.123413+0000","last_fullsized":"2024-03-28T15:01:37.123413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461324+0000","last_change":"2024-03-28T15:01:32.114975+0000","last_active":"2024-03-28T15:01:33.461324+0000","last_peered":"2024-03-28T15:01:33.461324+0000","last_clean":"2024-03-28T15:01:33.461324+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:01:33.461324+0000","last_undegraded":"2024-03-28T15:01:33.461324+0000","last_fullsized":"2024-03-28T15:01:33.461324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.482725+0000","last_change":"2024-03-28T15:01:31.097901+0000","last_active":"2024-03-28T15:01:34.482725+0000","last_peered":"2024-03-28T15:01:34.482725+0000","last_clean":"2024-03-28T15:01:34.482725+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:01:34.482725+0000","last_undegraded":"2024-03-28T15:01:34.482725+0000","last_fullsized":"2024-03-28T15:01:34.482725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108151+0000","last_change":"2024-03-28T15:01:32.120646+0000","last_active":"2024-03-28T15:01:33.108151+0000","last_peered":"2024-03-28T15:01:33.108151+0000","last_clean":"2024-03-28T15:01:33.108151+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:01:33.108151+0000","last_undegraded":"2024-03-28T15:01:33.108151+0000","last_fullsized":"2024-03-28T15:01:33.108151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460197+0000","last_change":"2024-03-28T15:01:31.086634+0000","last_active":"2024-03-28T15:01:33.460197+0000","last_peered":"2024-03-28T15:01:33.460197+0000","last_clean":"2024-03-28T15:01:33.460197+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:01:33.460197+0000","last_undegraded":"2024-03-28T15:01:33.460197+0000","last_fullsized":"2024-03-28T15:01:33.460197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461104+0000","last_change":"2024-03-28T15:01:32.112493+0000","last_active":"2024-03-28T15:01:33.461104+0000","last_peered":"2024-03-28T15:01:33.461104+0000","last_clean":"2024-03-28T15:01:33.461104+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:01:33.461104+0000","last_undegraded":"2024-03-28T15:01:33.461104+0000","last_fullsized":"2024-03-28T15:01:33.461104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461052+0000","last_change":"2024-03-28T15:01:20.724163+0000","last_active":"2024-03-28T15:01:33.461052+0000","last_peered":"2024-03-28T15:01:33.461052+0000","last_clean":"2024-03-28T15:01:33.461052+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:01:33.461052+0000","last_undegraded":"2024-03-28T15:01:33.461052+0000","last_fullsized":"2024-03-28T15:01:33.461052+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_clean_scrub_stamp":"2024-03-28T15:01:19.548090+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:15.791893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107058+0000","last_change":"2024-03-28T15:01:31.085453+0000","last_active":"2024-03-28T15:01:33.107058+0000","last_peered":"2024-03-28T15:01:33.107058+0000","last_clean":"2024-03-28T15:01:33.107058+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:01:33.107058+0000","last_undegraded":"2024-03-28T15:01:33.107058+0000","last_fullsized":"2024-03-28T15:01:33.107058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460326+0000","last_change":"2024-03-28T15:01:32.118678+0000","last_active":"2024-03-28T15:01:33.460326+0000","last_peered":"2024-03-28T15:01:33.460326+0000","last_clean":"2024-03-28T15:01:33.460326+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:01:33.460326+0000","last_undegraded":"2024-03-28T15:01:33.460326+0000","last_fullsized":"2024-03-28T15:01:33.460326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461116+0000","last_change":"2024-03-28T15:01:31.095439+0000","last_active":"2024-03-28T15:01:33.461116+0000","last_peered":"2024-03-28T15:01:33.461116+0000","last_clean":"2024-03-28T15:01:33.461116+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:01:33.461116+0000","last_undegraded":"2024-03-28T15:01:33.461116+0000","last_fullsized":"2024-03-28T15:01:33.461116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460849+0000","last_change":"2024-03-28T15:01:32.122655+0000","last_active":"2024-03-28T15:01:33.460849+0000","last_peered":"2024-03-28T15:01:33.460849+0000","last_clean":"2024-03-28T15:01:33.460849+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:01:33.460849+0000","last_undegraded":"2024-03-28T15:01:33.460849+0000","last_fullsized":"2024-03-28T15:01:33.460849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:05.942532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460385+0000","last_change":"2024-03-28T15:01:31.093265+0000","last_active":"2024-03-28T15:01:33.460385+0000","last_peered":"2024-03-28T15:01:33.460385+0000","last_clean":"2024-03-28T15:01:33.460385+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:01:33.460385+0000","last_undegraded":"2024-03-28T15:01:33.460385+0000","last_fullsized":"2024-03-28T15:01:33.460385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104892+0000","last_change":"2024-03-28T15:01:32.108597+0000","last_active":"2024-03-28T15:01:33.104892+0000","last_peered":"2024-03-28T15:01:33.104892+0000","last_clean":"2024-03-28T15:01:33.104892+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:01:33.104892+0000","last_undegraded":"2024-03-28T15:01:33.104892+0000","last_fullsized":"2024-03-28T15:01:33.104892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108209+0000","last_change":"2024-03-28T15:01:31.094750+0000","last_active":"2024-03-28T15:01:33.108209+0000","last_peered":"2024-03-28T15:01:33.108209+0000","last_clean":"2024-03-28T15:01:33.108209+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:01:33.108209+0000","last_undegraded":"2024-03-28T15:01:33.108209+0000","last_fullsized":"2024-03-28T15:01:33.108209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106970+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.106970+0000","last_peered":"2024-03-28T15:01:33.106970+0000","last_clean":"2024-03-28T15:01:33.106970+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:01:33.106970+0000","last_undegraded":"2024-03-28T15:01:33.106970+0000","last_fullsized":"2024-03-28T15:01:33.106970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:24.747778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104874+0000","last_change":"2024-03-28T15:01:31.093706+0000","last_active":"2024-03-28T15:01:33.104874+0000","last_peered":"2024-03-28T15:01:33.104874+0000","last_clean":"2024-03-28T15:01:33.104874+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:01:33.104874+0000","last_undegraded":"2024-03-28T15:01:33.104874+0000","last_fullsized":"2024-03-28T15:01:33.104874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108112+0000","last_change":"2024-03-28T15:01:32.122840+0000","last_active":"2024-03-28T15:01:33.108112+0000","last_peered":"2024-03-28T15:01:33.108112+0000","last_clean":"2024-03-28T15:01:33.108112+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:01:33.108112+0000","last_undegraded":"2024-03-28T15:01:33.108112+0000","last_fullsized":"2024-03-28T15:01:33.108112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461371+0000","last_change":"2024-03-28T15:01:31.095126+0000","last_active":"2024-03-28T15:01:33.461371+0000","last_peered":"2024-03-28T15:01:33.461371+0000","last_clean":"2024-03-28T15:01:33.461371+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:01:33.461371+0000","last_undegraded":"2024-03-28T15:01:33.461371+0000","last_fullsized":"2024-03-28T15:01:33.461371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:21.669535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104885+0000","last_change":"2024-03-28T15:01:32.111925+0000","last_active":"2024-03-28T15:01:33.104885+0000","last_peered":"2024-03-28T15:01:33.104885+0000","last_clean":"2024-03-28T15:01:33.104885+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:01:33.104885+0000","last_undegraded":"2024-03-28T15:01:33.104885+0000","last_fullsized":"2024-03-28T15:01:33.104885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.482739+0000","last_change":"2024-03-28T15:01:31.093197+0000","last_active":"2024-03-28T15:01:34.482739+0000","last_peered":"2024-03-28T15:01:34.482739+0000","last_clean":"2024-03-28T15:01:34.482739+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:01:34.482739+0000","last_undegraded":"2024-03-28T15:01:34.482739+0000","last_fullsized":"2024-03-28T15:01:34.482739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461153+0000","last_change":"2024-03-28T15:01:32.115432+0000","last_active":"2024-03-28T15:01:33.461153+0000","last_peered":"2024-03-28T15:01:33.461153+0000","last_clean":"2024-03-28T15:01:33.461153+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:01:33.461153+0000","last_undegraded":"2024-03-28T15:01:33.461153+0000","last_fullsized":"2024-03-28T15:01:33.461153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.125948+0000","last_change":"2024-03-28T15:01:31.088204+0000","last_active":"2024-03-28T15:01:37.125948+0000","last_peered":"2024-03-28T15:01:37.125948+0000","last_clean":"2024-03-28T15:01:37.125948+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:01:37.125948+0000","last_undegraded":"2024-03-28T15:01:37.125948+0000","last_fullsized":"2024-03-28T15:01:37.125948+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461225+0000","last_change":"2024-03-28T15:01:32.114331+0000","last_active":"2024-03-28T15:01:33.461225+0000","last_peered":"2024-03-28T15:01:33.461225+0000","last_clean":"2024-03-28T15:01:33.461225+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:01:33.461225+0000","last_undegraded":"2024-03-28T15:01:33.461225+0000","last_fullsized":"2024-03-28T15:01:33.461225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460969+0000","last_change":"2024-03-28T15:01:31.095371+0000","last_active":"2024-03-28T15:01:33.460969+0000","last_peered":"2024-03-28T15:01:33.460969+0000","last_clean":"2024-03-28T15:01:33.460969+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:01:33.460969+0000","last_undegraded":"2024-03-28T15:01:33.460969+0000","last_fullsized":"2024-03-28T15:01:33.460969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107409+0000","last_change":"2024-03-28T15:01:32.110950+0000","last_active":"2024-03-28T15:01:33.107409+0000","last_peered":"2024-03-28T15:01:33.107409+0000","last_clean":"2024-03-28T15:01:33.107409+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:01:33.107409+0000","last_undegraded":"2024-03-28T15:01:33.107409+0000","last_fullsized":"2024-03-28T15:01:33.107409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105009+0000","last_change":"2024-03-28T15:01:31.094542+0000","last_active":"2024-03-28T15:01:33.105009+0000","last_peered":"2024-03-28T15:01:33.105009+0000","last_clean":"2024-03-28T15:01:33.105009+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:01:33.105009+0000","last_undegraded":"2024-03-28T15:01:33.105009+0000","last_fullsized":"2024-03-28T15:01:33.105009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460466+0000","last_change":"2024-03-28T15:01:32.112158+0000","last_active":"2024-03-28T15:01:33.460466+0000","last_peered":"2024-03-28T15:01:33.460466+0000","last_clean":"2024-03-28T15:01:33.460466+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:01:33.460466+0000","last_undegraded":"2024-03-28T15:01:33.460466+0000","last_fullsized":"2024-03-28T15:01:33.460466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104884+0000","last_change":"2024-03-28T15:01:31.094440+0000","last_active":"2024-03-28T15:01:33.104884+0000","last_peered":"2024-03-28T15:01:33.104884+0000","last_clean":"2024-03-28T15:01:33.104884+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:01:33.104884+0000","last_undegraded":"2024-03-28T15:01:33.104884+0000","last_fullsized":"2024-03-28T15:01:33.104884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460758+0000","last_change":"2024-03-28T15:01:32.115192+0000","last_active":"2024-03-28T15:01:33.460758+0000","last_peered":"2024-03-28T15:01:33.460758+0000","last_clean":"2024-03-28T15:01:33.460758+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:01:33.460758+0000","last_undegraded":"2024-03-28T15:01:33.460758+0000","last_fullsized":"2024-03-28T15:01:33.460758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.125967+0000","last_change":"2024-03-28T15:01:31.096352+0000","last_active":"2024-03-28T15:01:37.125967+0000","last_peered":"2024-03-28T15:01:37.125967+0000","last_clean":"2024-03-28T15:01:37.125967+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:01:37.125967+0000","last_undegraded":"2024-03-28T15:01:37.125967+0000","last_fullsized":"2024-03-28T15:01:37.125967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460555+0000","last_change":"2024-03-28T15:01:32.124700+0000","last_active":"2024-03-28T15:01:33.460555+0000","last_peered":"2024-03-28T15:01:33.460555+0000","last_clean":"2024-03-28T15:01:33.460555+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:01:33.460555+0000","last_undegraded":"2024-03-28T15:01:33.460555+0000","last_fullsized":"2024-03-28T15:01:33.460555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107999+0000","last_change":"2024-03-28T15:01:32.123638+0000","last_active":"2024-03-28T15:01:33.107999+0000","last_peered":"2024-03-28T15:01:33.107999+0000","last_clean":"2024-03-28T15:01:33.107999+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:01:33.107999+0000","last_undegraded":"2024-03-28T15:01:33.107999+0000","last_fullsized":"2024-03-28T15:01:33.107999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.121539+0000","last_change":"2024-03-28T15:01:31.094281+0000","last_active":"2024-03-28T15:01:37.121539+0000","last_peered":"2024-03-28T15:01:37.121539+0000","last_clean":"2024-03-28T15:01:37.121539+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:01:37.121539+0000","last_undegraded":"2024-03-28T15:01:37.121539+0000","last_fullsized":"2024-03-28T15:01:37.121539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107988+0000","last_change":"2024-03-28T15:01:32.112496+0000","last_active":"2024-03-28T15:01:33.107988+0000","last_peered":"2024-03-28T15:01:33.107988+0000","last_clean":"2024-03-28T15:01:33.107988+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:01:33.107988+0000","last_undegraded":"2024-03-28T15:01:33.107988+0000","last_fullsized":"2024-03-28T15:01:33.107988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126177+0000","last_change":"2024-03-28T15:01:31.093773+0000","last_active":"2024-03-28T15:01:37.126177+0000","last_peered":"2024-03-28T15:01:37.126177+0000","last_clean":"2024-03-28T15:01:37.126177+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:01:37.126177+0000","last_undegraded":"2024-03-28T15:01:37.126177+0000","last_fullsized":"2024-03-28T15:01:37.126177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:54.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460976+0000","last_change":"2024-03-28T15:01:32.117275+0000","last_active":"2024-03-28T15:01:33.460976+0000","last_peered":"2024-03-28T15:01:33.460976+0000","last_clean":"2024-03-28T15:01:33.460976+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:01:33.460976+0000","last_undegraded":"2024-03-28T15:01:33.460976+0000","last_fullsized":"2024-03-28T15:01:33.460976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.475554+0000","last_change":"2024-03-28T15:01:31.096725+0000","last_active":"2024-03-28T15:01:34.475554+0000","last_peered":"2024-03-28T15:01:34.475554+0000","last_clean":"2024-03-28T15:01:34.475554+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:01:34.475554+0000","last_undegraded":"2024-03-28T15:01:34.475554+0000","last_fullsized":"2024-03-28T15:01:34.475554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460734+0000","last_change":"2024-03-28T15:01:32.120071+0000","last_active":"2024-03-28T15:01:33.460734+0000","last_peered":"2024-03-28T15:01:33.460734+0000","last_clean":"2024-03-28T15:01:33.460734+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:01:33.460734+0000","last_undegraded":"2024-03-28T15:01:33.460734+0000","last_fullsized":"2024-03-28T15:01:33.460734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126217+0000","last_change":"2024-03-28T15:01:31.095351+0000","last_active":"2024-03-28T15:01:37.126217+0000","last_peered":"2024-03-28T15:01:37.126217+0000","last_clean":"2024-03-28T15:01:37.126217+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:01:37.126217+0000","last_undegraded":"2024-03-28T15:01:37.126217+0000","last_fullsized":"2024-03-28T15:01:37.126217+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":73802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":73802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:01:51.564 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-03-28T15:01:51.564 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:01:51.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.701+0000 7fafcf58a640 1 Processor -- start 2024-03-28T15:01:51.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.701+0000 7fafcf58a640 1 -- start start 2024-03-28T15:01:51.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.702+0000 7fafcf58a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc8151990 0x7fafc80c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.702+0000 7fafcf58a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc806caa0 con 0x7fafc80c2d00 2024-03-28T15:01:51.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.702+0000 7fafcf58a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc806cd80 con 0x7fafc80b1e30 2024-03-28T15:01:51.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.702+0000 7fafcf58a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fafc80c1da0 con 0x7fafc8151990 2024-03-28T15:01:51.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.702+0000 7fafcdb27640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fafc80c2d00 0x7fafc8161dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49790/0 (socket says 172.21.15.186:49790) 2024-03-28T15:01:51.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.702+0000 7fafcdb27640 1 -- 172.21.15.186:0/2298665655 learned_addr learned my addr 172.21.15.186:0/2298665655 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:51.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.702+0000 7fafcd326640 1 --2- 172.21.15.186:0/2298665655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc8151990 0x7fafc80c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.702+0000 7fafcd326640 1 -- 172.21.15.186:0/2298665655 >> v1:172.21.15.191:6789/0 conn(0x7fafc80b1e30 legacy=0x7fafc80712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.702+0000 7fafcd326640 1 -- 172.21.15.186:0/2298665655 >> v1:172.21.15.186:6789/0 conn(0x7fafc80c2d00 legacy=0x7fafc8161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.702+0000 7fafbffff640 1 -- 172.21.15.186:0/2298665655 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340564507 0 0) 0x7fafc806caa0 con 0x7fafc80c2d00 2024-03-28T15:01:51.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.702+0000 7fafcd326640 1 -- 172.21.15.186:0/2298665655 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc80c1340 con 0x7fafc8151990 2024-03-28T15:01:51.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.703+0000 7fafcd326640 1 --2- 172.21.15.186:0/2298665655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc8151990 0x7fafc80c2230 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fafc4009750 tx=0x7fafc402b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efa3aca286aad66b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:51.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.703+0000 7fafbffff640 1 -- 172.21.15.186:0/2298665655 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4038070 con 0x7fafc8151990 2024-03-28T15:01:51.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.703+0000 7fafbffff640 1 -- 172.21.15.186:0/2298665655 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafc4031710 con 0x7fafc8151990 2024-03-28T15:01:51.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.703+0000 7fafbffff640 1 -- 172.21.15.186:0/2298665655 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4031ae0 con 0x7fafc8151990 2024-03-28T15:01:51.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.704+0000 7fafcf58a640 1 -- 172.21.15.186:0/2298665655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc8151990 msgr2=0x7fafc80c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.704+0000 7fafcf58a640 1 --2- 172.21.15.186:0/2298665655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc8151990 0x7fafc80c2230 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fafc4009750 tx=0x7fafc402b7b0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.704+0000 7fafcf58a640 1 -- 172.21.15.186:0/2298665655 shutdown_connections 2024-03-28T15:01:51.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.704+0000 7fafcf58a640 1 --2- 172.21.15.186:0/2298665655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc8151990 0x7fafc80c2230 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.704+0000 7fafcf58a640 1 -- 172.21.15.186:0/2298665655 >> 172.21.15.186:0/2298665655 conn(0x7fafc80c8a20 msgr2=0x7fafc80c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:51.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.704+0000 7fafcf58a640 1 -- 172.21.15.186:0/2298665655 shutdown_connections 2024-03-28T15:01:51.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.704+0000 7fafcf58a640 1 -- 172.21.15.186:0/2298665655 wait complete. 2024-03-28T15:01:51.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.705+0000 7fafcf58a640 1 Processor -- start 2024-03-28T15:01:51.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.705+0000 7fafcf58a640 1 -- start start 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.705+0000 7fafcf58a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80b1e30 0x7fafc80c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.705+0000 7fafcf58a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc80c2d00 0x7fafc81ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.705+0000 7fafcf58a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafc8151990 0x7fafc8154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.705+0000 7fafcf58a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fafc8073dd0 con 0x7fafc8151990 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.705+0000 7fafcf58a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fafc80c1900 con 0x7fafc80c2d00 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.705+0000 7fafcf58a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fafc8073f50 con 0x7fafc80b1e30 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcdb27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafc8151990 0x7fafc8154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcdb27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafc8151990 0x7fafc8154850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42398/0 (socket says 172.21.15.186:42398) 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcdb27640 1 -- 172.21.15.186:0/1188145909 learned_addr learned my addr 172.21.15.186:0/1188145909 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcd326640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80b1e30 0x7fafc80c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafccb25640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc80c2d00 0x7fafc81ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcd326640 1 -- 172.21.15.186:0/1188145909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc80c2d00 msgr2=0x7fafc81ee6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcd326640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc80c2d00 0x7fafc81ee6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcd326640 1 -- 172.21.15.186:0/1188145909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafc8151990 msgr2=0x7fafc8154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcd326640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafc8151990 0x7fafc8154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcd326640 1 -- 172.21.15.186:0/1188145909 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc80c4b60 con 0x7fafc80b1e30 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcdb27640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafc8151990 0x7fafc8154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcd326640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80b1e30 0x7fafc80c41f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fafc806e690 tx=0x7fafc40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafbdffb640 1 -- 172.21.15.186:0/1188145909 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4043070 con 0x7fafc80b1e30 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafbdffb640 1 -- 172.21.15.186:0/1188145909 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafc40339c0 con 0x7fafc80b1e30 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafcf58a640 1 -- 172.21.15.186:0/1188145909 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc80c6190 con 0x7fafc80b1e30 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.706+0000 7fafbdffb640 1 -- 172.21.15.186:0/1188145909 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4038040 con 0x7fafc80b1e30 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.707+0000 7fafcf58a640 1 -- 172.21.15.186:0/1188145909 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc814dfb0 con 0x7fafc80b1e30 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.707+0000 7fafcf58a640 1 -- 172.21.15.186:0/1188145909 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc80c79b0 con 0x7fafc80b1e30 2024-03-28T15:01:51.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.708+0000 7fafbdffb640 1 -- 172.21.15.186:0/1188145909 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fafc4046070 con 0x7fafc80b1e30 2024-03-28T15:01:51.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.708+0000 7fafbdffb640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fafac06c530 0x7fafac06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:51.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.708+0000 7fafbdffb640 1 -- 172.21.15.186:0/1188145909 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fafc40ad610 con 0x7fafc80b1e30 2024-03-28T15:01:51.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.708+0000 7fafccb25640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fafac06c530 0x7fafac06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:51.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.709+0000 7fafccb25640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fafac06c530 0x7fafac06e9f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fafc8151fa0 tx=0x7fafb8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:51.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.713+0000 7fafbdffb640 1 -- 172.21.15.186:0/1188145909 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fafc80c79b0 con 0x7fafc80b1e30 2024-03-28T15:01:51.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.885+0000 7fafcf58a640 1 -- 172.21.15.186:0/1188145909 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fafc8072040 con 0x7fafac06c530 2024-03-28T15:01:51.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.891+0000 7fafbdffb640 1 -- 172.21.15.186:0/1188145909 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251831 (secure 0 0 0) 0x7fafc8072040 con 0x7fafac06c530 2024-03-28T15:01:51.893 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:51.895 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:01:51.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 -- 172.21.15.186:0/1188145909 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fafac06c530 msgr2=0x7fafac06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fafac06c530 0x7fafac06e9f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fafc8151fa0 tx=0x7fafb8009040 comp rx=0 tx=0).stop 2024-03-28T15:01:51.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 -- 172.21.15.186:0/1188145909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80b1e30 msgr2=0x7fafc80c41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:51.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80b1e30 0x7fafc80c41f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fafc806e690 tx=0x7fafc40046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 -- 172.21.15.186:0/1188145909 shutdown_connections 2024-03-28T15:01:51.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fafac06c530 0x7fafac06e9f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafc8151990 0x7fafc8154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc80c2d00 0x7fafc81ee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 --2- 172.21.15.186:0/1188145909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80b1e30 0x7fafc80c41f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:51.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 -- 172.21.15.186:0/1188145909 >> 172.21.15.186:0/1188145909 conn(0x7fafc80c8a20 msgr2=0x7fafc80713f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:51.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 -- 172.21.15.186:0/1188145909 shutdown_connections 2024-03-28T15:01:51.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:51.894+0000 7fafcf58a640 1 -- 172.21.15.186:0/1188145909 wait complete. 2024-03-28T15:01:51.905 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-28T15:01:50.638678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594034,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218952,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755768,"statfs":{"total":773094113280,"available":772869906432,"internally_reserved":0,"allocated":2891776,"data_stored":1755540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1256,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002386"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461304+0000","last_change":"2024-03-28T15:01:32.122226+0000","last_active":"2024-03-28T15:01:33.461304+0000","last_peered":"2024-03-28T15:01:33.461304+0000","last_clean":"2024-03-28T15:01:33.461304+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:01:33.461304+0000","last_undegraded":"2024-03-28T15:01:33.461304+0000","last_fullsized":"2024-03-28T15:01:33.461304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104984+0000","last_change":"2024-03-28T15:01:32.115850+0000","last_active":"2024-03-28T15:01:33.104984+0000","last_peered":"2024-03-28T15:01:33.104984+0000","last_clean":"2024-03-28T15:01:33.104984+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:01:33.104984+0000","last_undegraded":"2024-03-28T15:01:33.104984+0000","last_fullsized":"2024-03-28T15:01:33.104984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.119744+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:57.022614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107952+0000","last_change":"2024-03-28T15:01:32.120916+0000","last_active":"2024-03-28T15:01:33.107952+0000","last_peered":"2024-03-28T15:01:33.107952+0000","last_clean":"2024-03-28T15:01:33.107952+0000","last_became_active":"2024-03-28T15:01:32.117449+0000","last_became_peered":"2024-03-28T15:01:32.117449+0000","last_unstale":"2024-03-28T15:01:33.107952+0000","last_undegraded":"2024-03-28T15:01:33.107952+0000","last_fullsized":"2024-03-28T15:01:33.107952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461016+0000","last_change":"2024-03-28T15:01:32.112835+0000","last_active":"2024-03-28T15:01:33.461016+0000","last_peered":"2024-03-28T15:01:33.461016+0000","last_clean":"2024-03-28T15:01:33.461016+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:01:33.461016+0000","last_undegraded":"2024-03-28T15:01:33.461016+0000","last_fullsized":"2024-03-28T15:01:33.461016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107714+0000","last_change":"2024-03-28T15:01:32.116248+0000","last_active":"2024-03-28T15:01:33.107714+0000","last_peered":"2024-03-28T15:01:33.107714+0000","last_clean":"2024-03-28T15:01:33.107714+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:01:33.107714+0000","last_undegraded":"2024-03-28T15:01:33.107714+0000","last_fullsized":"2024-03-28T15:01:33.107714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460486+0000","last_change":"2024-03-28T15:01:32.115221+0000","last_active":"2024-03-28T15:01:33.460486+0000","last_peered":"2024-03-28T15:01:33.460486+0000","last_clean":"2024-03-28T15:01:33.460486+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:01:33.460486+0000","last_undegraded":"2024-03-28T15:01:33.460486+0000","last_fullsized":"2024-03-28T15:01:33.460486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460218+0000","last_change":"2024-03-28T15:01:32.118310+0000","last_active":"2024-03-28T15:01:33.460218+0000","last_peered":"2024-03-28T15:01:33.460218+0000","last_clean":"2024-03-28T15:01:33.460218+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:01:33.460218+0000","last_undegraded":"2024-03-28T15:01:33.460218+0000","last_fullsized":"2024-03-28T15:01:33.460218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461212+0000","last_change":"2024-03-28T15:01:32.114205+0000","last_active":"2024-03-28T15:01:33.461212+0000","last_peered":"2024-03-28T15:01:33.461212+0000","last_clean":"2024-03-28T15:01:33.461212+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:01:33.461212+0000","last_undegraded":"2024-03-28T15:01:33.461212+0000","last_fullsized":"2024-03-28T15:01:33.461212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460905+0000","last_change":"2024-03-28T15:01:32.125178+0000","last_active":"2024-03-28T15:01:33.460905+0000","last_peered":"2024-03-28T15:01:33.460905+0000","last_clean":"2024-03-28T15:01:33.460905+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:01:33.460905+0000","last_undegraded":"2024-03-28T15:01:33.460905+0000","last_fullsized":"2024-03-28T15:01:33.460905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460676+0000","last_change":"2024-03-28T15:01:32.125839+0000","last_active":"2024-03-28T15:01:33.460676+0000","last_peered":"2024-03-28T15:01:33.460676+0000","last_clean":"2024-03-28T15:01:33.460676+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:01:33.460676+0000","last_undegraded":"2024-03-28T15:01:33.460676+0000","last_fullsized":"2024-03-28T15:01:33.460676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107231+0000","last_change":"2024-03-28T15:01:32.121874+0000","last_active":"2024-03-28T15:01:33.107231+0000","last_peered":"2024-03-28T15:01:33.107231+0000","last_clean":"2024-03-28T15:01:33.107231+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:01:33.107231+0000","last_undegraded":"2024-03-28T15:01:33.107231+0000","last_fullsized":"2024-03-28T15:01:33.107231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:20.744143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460735+0000","last_change":"2024-03-28T15:01:32.119348+0000","last_active":"2024-03-28T15:01:33.460735+0000","last_peered":"2024-03-28T15:01:33.460735+0000","last_clean":"2024-03-28T15:01:33.460735+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:01:33.460735+0000","last_undegraded":"2024-03-28T15:01:33.460735+0000","last_fullsized":"2024-03-28T15:01:33.460735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461204+0000","last_change":"2024-03-28T15:01:32.114290+0000","last_active":"2024-03-28T15:01:33.461204+0000","last_peered":"2024-03-28T15:01:33.461204+0000","last_clean":"2024-03-28T15:01:33.461204+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:01:33.461204+0000","last_undegraded":"2024-03-28T15:01:33.461204+0000","last_fullsized":"2024-03-28T15:01:33.461204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105071+0000","last_change":"2024-03-28T15:01:32.113654+0000","last_active":"2024-03-28T15:01:33.105071+0000","last_peered":"2024-03-28T15:01:33.105071+0000","last_clean":"2024-03-28T15:01:33.105071+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:01:33.105071+0000","last_undegraded":"2024-03-28T15:01:33.105071+0000","last_fullsized":"2024-03-28T15:01:33.105071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107620+0000","last_change":"2024-03-28T15:01:32.115901+0000","last_active":"2024-03-28T15:01:33.107620+0000","last_peered":"2024-03-28T15:01:33.107620+0000","last_clean":"2024-03-28T15:01:33.107620+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:01:33.107620+0000","last_undegraded":"2024-03-28T15:01:33.107620+0000","last_fullsized":"2024-03-28T15:01:33.107620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108153+0000","last_change":"2024-03-28T15:01:32.113156+0000","last_active":"2024-03-28T15:01:33.108153+0000","last_peered":"2024-03-28T15:01:33.108153+0000","last_clean":"2024-03-28T15:01:33.108153+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:01:33.108153+0000","last_undegraded":"2024-03-28T15:01:33.108153+0000","last_fullsized":"2024-03-28T15:01:33.108153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:22:14.792159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460557+0000","last_change":"2024-03-28T15:01:32.119785+0000","last_active":"2024-03-28T15:01:33.460557+0000","last_peered":"2024-03-28T15:01:33.460557+0000","last_clean":"2024-03-28T15:01:33.460557+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:01:33.460557+0000","last_undegraded":"2024-03-28T15:01:33.460557+0000","last_fullsized":"2024-03-28T15:01:33.460557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461426+0000","last_change":"2024-03-28T15:01:32.111912+0000","last_active":"2024-03-28T15:01:33.461426+0000","last_peered":"2024-03-28T15:01:33.461426+0000","last_clean":"2024-03-28T15:01:33.461426+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:01:33.461426+0000","last_undegraded":"2024-03-28T15:01:33.461426+0000","last_fullsized":"2024-03-28T15:01:33.461426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:52.632729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461025+0000","last_change":"2024-03-28T15:01:32.124381+0000","last_active":"2024-03-28T15:01:33.461025+0000","last_peered":"2024-03-28T15:01:33.461025+0000","last_clean":"2024-03-28T15:01:33.461025+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:01:33.461025+0000","last_undegraded":"2024-03-28T15:01:33.461025+0000","last_fullsized":"2024-03-28T15:01:33.461025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.120110+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107929+0000","last_change":"2024-03-28T15:01:32.113699+0000","last_active":"2024-03-28T15:01:33.107929+0000","last_peered":"2024-03-28T15:01:33.107929+0000","last_clean":"2024-03-28T15:01:33.107929+0000","last_became_active":"2024-03-28T15:01:32.113321+0000","last_became_peered":"2024-03-28T15:01:32.113321+0000","last_unstale":"2024-03-28T15:01:33.107929+0000","last_undegraded":"2024-03-28T15:01:33.107929+0000","last_fullsized":"2024-03-28T15:01:33.107929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460506+0000","last_change":"2024-03-28T15:01:32.113345+0000","last_active":"2024-03-28T15:01:33.460506+0000","last_peered":"2024-03-28T15:01:33.460506+0000","last_clean":"2024-03-28T15:01:33.460506+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:01:33.460506+0000","last_undegraded":"2024-03-28T15:01:33.460506+0000","last_fullsized":"2024-03-28T15:01:33.460506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107235+0000","last_change":"2024-03-28T15:01:32.112328+0000","last_active":"2024-03-28T15:01:33.107235+0000","last_peered":"2024-03-28T15:01:33.107235+0000","last_clean":"2024-03-28T15:01:33.107235+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:01:33.107235+0000","last_undegraded":"2024-03-28T15:01:33.107235+0000","last_fullsized":"2024-03-28T15:01:33.107235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461017+0000","last_change":"2024-03-28T15:01:32.122399+0000","last_active":"2024-03-28T15:01:33.461017+0000","last_peered":"2024-03-28T15:01:33.461017+0000","last_clean":"2024-03-28T15:01:33.461017+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:01:33.461017+0000","last_undegraded":"2024-03-28T15:01:33.461017+0000","last_fullsized":"2024-03-28T15:01:33.461017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460971+0000","last_change":"2024-03-28T15:01:32.123879+0000","last_active":"2024-03-28T15:01:33.460971+0000","last_peered":"2024-03-28T15:01:33.460971+0000","last_clean":"2024-03-28T15:01:33.460971+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:01:33.460971+0000","last_undegraded":"2024-03-28T15:01:33.460971+0000","last_fullsized":"2024-03-28T15:01:33.460971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104923+0000","last_change":"2024-03-28T15:01:32.110380+0000","last_active":"2024-03-28T15:01:33.104923+0000","last_peered":"2024-03-28T15:01:33.104923+0000","last_clean":"2024-03-28T15:01:33.104923+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:01:33.104923+0000","last_undegraded":"2024-03-28T15:01:33.104923+0000","last_fullsized":"2024-03-28T15:01:33.104923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460754+0000","last_change":"2024-03-28T15:01:32.113422+0000","last_active":"2024-03-28T15:01:33.460754+0000","last_peered":"2024-03-28T15:01:33.460754+0000","last_clean":"2024-03-28T15:01:33.460754+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:01:33.460754+0000","last_undegraded":"2024-03-28T15:01:33.460754+0000","last_fullsized":"2024-03-28T15:01:33.460754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461261+0000","last_change":"2024-03-28T15:01:32.117560+0000","last_active":"2024-03-28T15:01:33.461261+0000","last_peered":"2024-03-28T15:01:33.461261+0000","last_clean":"2024-03-28T15:01:33.461261+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:01:33.461261+0000","last_undegraded":"2024-03-28T15:01:33.461261+0000","last_fullsized":"2024-03-28T15:01:33.461261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108157+0000","last_change":"2024-03-28T15:01:32.117440+0000","last_active":"2024-03-28T15:01:33.108157+0000","last_peered":"2024-03-28T15:01:33.108157+0000","last_clean":"2024-03-28T15:01:33.108157+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:01:33.108157+0000","last_undegraded":"2024-03-28T15:01:33.108157+0000","last_fullsized":"2024-03-28T15:01:33.108157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460388+0000","last_change":"2024-03-28T15:01:32.111344+0000","last_active":"2024-03-28T15:01:33.460388+0000","last_peered":"2024-03-28T15:01:33.460388+0000","last_clean":"2024-03-28T15:01:33.460388+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:01:33.460388+0000","last_undegraded":"2024-03-28T15:01:33.460388+0000","last_fullsized":"2024-03-28T15:01:33.460388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460295+0000","last_change":"2024-03-28T15:01:32.126811+0000","last_active":"2024-03-28T15:01:33.460295+0000","last_peered":"2024-03-28T15:01:33.460295+0000","last_clean":"2024-03-28T15:01:33.460295+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:01:33.460295+0000","last_undegraded":"2024-03-28T15:01:33.460295+0000","last_fullsized":"2024-03-28T15:01:33.460295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106843+0000","last_change":"2024-03-28T15:01:32.112050+0000","last_active":"2024-03-28T15:01:33.106843+0000","last_peered":"2024-03-28T15:01:33.106843+0000","last_clean":"2024-03-28T15:01:33.106843+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:01:33.106843+0000","last_undegraded":"2024-03-28T15:01:33.106843+0000","last_fullsized":"2024-03-28T15:01:33.106843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:30.376545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.475527+0000","last_change":"2024-03-28T15:01:31.097415+0000","last_active":"2024-03-28T15:01:34.475527+0000","last_peered":"2024-03-28T15:01:34.475527+0000","last_clean":"2024-03-28T15:01:34.475527+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:01:34.475527+0000","last_undegraded":"2024-03-28T15:01:34.475527+0000","last_fullsized":"2024-03-28T15:01:34.475527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461379+0000","last_change":"2024-03-28T15:01:32.123479+0000","last_active":"2024-03-28T15:01:33.461379+0000","last_peered":"2024-03-28T15:01:33.461379+0000","last_clean":"2024-03-28T15:01:33.461379+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:01:33.461379+0000","last_undegraded":"2024-03-28T15:01:33.461379+0000","last_fullsized":"2024-03-28T15:01:33.461379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:35.502796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.476969+0000","last_change":"2024-03-28T15:01:31.088437+0000","last_active":"2024-03-28T15:01:34.476969+0000","last_peered":"2024-03-28T15:01:34.476969+0000","last_clean":"2024-03-28T15:01:34.476969+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:01:34.476969+0000","last_undegraded":"2024-03-28T15:01:34.476969+0000","last_fullsized":"2024-03-28T15:01:34.476969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460403+0000","last_change":"2024-03-28T15:01:31.097499+0000","last_active":"2024-03-28T15:01:33.460403+0000","last_peered":"2024-03-28T15:01:33.460403+0000","last_clean":"2024-03-28T15:01:33.460403+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:01:33.460403+0000","last_undegraded":"2024-03-28T15:01:33.460403+0000","last_fullsized":"2024-03-28T15:01:33.460403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460343+0000","last_change":"2024-03-28T15:01:32.115727+0000","last_active":"2024-03-28T15:01:33.460343+0000","last_peered":"2024-03-28T15:01:33.460343+0000","last_clean":"2024-03-28T15:01:33.460343+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:01:33.460343+0000","last_undegraded":"2024-03-28T15:01:33.460343+0000","last_fullsized":"2024-03-28T15:01:33.460343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107881+0000","last_change":"2024-03-28T15:01:31.086252+0000","last_active":"2024-03-28T15:01:33.107881+0000","last_peered":"2024-03-28T15:01:33.107881+0000","last_clean":"2024-03-28T15:01:33.107881+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:01:33.107881+0000","last_undegraded":"2024-03-28T15:01:33.107881+0000","last_fullsized":"2024-03-28T15:01:33.107881+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104868+0000","last_change":"2024-03-28T15:01:32.110465+0000","last_active":"2024-03-28T15:01:33.104868+0000","last_peered":"2024-03-28T15:01:33.104868+0000","last_clean":"2024-03-28T15:01:33.104868+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:01:33.104868+0000","last_undegraded":"2024-03-28T15:01:33.104868+0000","last_fullsized":"2024-03-28T15:01:33.104868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460446+0000","last_change":"2024-03-28T15:01:31.092338+0000","last_active":"2024-03-28T15:01:33.460446+0000","last_peered":"2024-03-28T15:01:33.460446+0000","last_clean":"2024-03-28T15:01:33.460446+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:01:33.460446+0000","last_undegraded":"2024-03-28T15:01:33.460446+0000","last_fullsized":"2024-03-28T15:01:33.460446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107369+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.107369+0000","last_peered":"2024-03-28T15:01:33.107369+0000","last_clean":"2024-03-28T15:01:33.107369+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:01:33.107369+0000","last_undegraded":"2024-03-28T15:01:33.107369+0000","last_fullsized":"2024-03-28T15:01:33.107369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:53.318777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107482+0000","last_change":"2024-03-28T15:01:31.088909+0000","last_active":"2024-03-28T15:01:33.107482+0000","last_peered":"2024-03-28T15:01:33.107482+0000","last_clean":"2024-03-28T15:01:33.107482+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:01:33.107482+0000","last_undegraded":"2024-03-28T15:01:33.107482+0000","last_fullsized":"2024-03-28T15:01:33.107482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461141+0000","last_change":"2024-03-28T15:01:32.117188+0000","last_active":"2024-03-28T15:01:33.461141+0000","last_peered":"2024-03-28T15:01:33.461141+0000","last_clean":"2024-03-28T15:01:33.461141+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:01:33.461141+0000","last_undegraded":"2024-03-28T15:01:33.461141+0000","last_fullsized":"2024-03-28T15:01:33.461141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.121351+0000","last_change":"2024-03-28T15:01:31.095935+0000","last_active":"2024-03-28T15:01:37.121351+0000","last_peered":"2024-03-28T15:01:37.121351+0000","last_clean":"2024-03-28T15:01:37.121351+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:01:37.121351+0000","last_undegraded":"2024-03-28T15:01:37.121351+0000","last_fullsized":"2024-03-28T15:01:37.121351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107512+0000","last_change":"2024-03-28T15:01:32.120677+0000","last_active":"2024-03-28T15:01:33.107512+0000","last_peered":"2024-03-28T15:01:33.107512+0000","last_clean":"2024-03-28T15:01:33.107512+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:01:33.107512+0000","last_undegraded":"2024-03-28T15:01:33.107512+0000","last_fullsized":"2024-03-28T15:01:33.107512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:55.008733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:47.705969+0000","last_change":"2024-03-28T15:01:31.089380+0000","last_active":"2024-03-28T15:01:47.705969+0000","last_peered":"2024-03-28T15:01:47.705969+0000","last_clean":"2024-03-28T15:01:47.705969+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:01:47.705969+0000","last_undegraded":"2024-03-28T15:01:47.705969+0000","last_fullsized":"2024-03-28T15:01:47.705969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460937+0000","last_change":"2024-03-28T15:01:32.110749+0000","last_active":"2024-03-28T15:01:33.460937+0000","last_peered":"2024-03-28T15:01:33.460937+0000","last_clean":"2024-03-28T15:01:33.460937+0000","last_became_active":"2024-03-28T15:01:32.110332+0000","last_became_peered":"2024-03-28T15:01:32.110332+0000","last_unstale":"2024-03-28T15:01:33.460937+0000","last_undegraded":"2024-03-28T15:01:33.460937+0000","last_fullsized":"2024-03-28T15:01:33.460937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.122627+0000","last_change":"2024-03-28T15:01:31.098172+0000","last_active":"2024-03-28T15:01:37.122627+0000","last_peered":"2024-03-28T15:01:37.122627+0000","last_clean":"2024-03-28T15:01:37.122627+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:01:37.122627+0000","last_undegraded":"2024-03-28T15:01:37.122627+0000","last_fullsized":"2024-03-28T15:01:37.122627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:16.015528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461045+0000","last_change":"2024-03-28T15:01:32.117690+0000","last_active":"2024-03-28T15:01:33.461045+0000","last_peered":"2024-03-28T15:01:33.461045+0000","last_clean":"2024-03-28T15:01:33.461045+0000","last_became_active":"2024-03-28T15:01:32.117412+0000","last_became_peered":"2024-03-28T15:01:32.117412+0000","last_unstale":"2024-03-28T15:01:33.461045+0000","last_undegraded":"2024-03-28T15:01:33.461045+0000","last_fullsized":"2024-03-28T15:01:33.461045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108124+0000","last_change":"2024-03-28T15:01:31.090519+0000","last_active":"2024-03-28T15:01:33.108124+0000","last_peered":"2024-03-28T15:01:33.108124+0000","last_clean":"2024-03-28T15:01:33.108124+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:01:33.108124+0000","last_undegraded":"2024-03-28T15:01:33.108124+0000","last_fullsized":"2024-03-28T15:01:33.108124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460932+0000","last_change":"2024-03-28T15:01:32.119827+0000","last_active":"2024-03-28T15:01:33.460932+0000","last_peered":"2024-03-28T15:01:33.460932+0000","last_clean":"2024-03-28T15:01:33.460932+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:01:33.460932+0000","last_undegraded":"2024-03-28T15:01:33.460932+0000","last_fullsized":"2024-03-28T15:01:33.460932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461234+0000","last_change":"2024-03-28T15:01:31.088364+0000","last_active":"2024-03-28T15:01:33.461234+0000","last_peered":"2024-03-28T15:01:33.461234+0000","last_clean":"2024-03-28T15:01:33.461234+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:01:33.461234+0000","last_undegraded":"2024-03-28T15:01:33.461234+0000","last_fullsized":"2024-03-28T15:01:33.461234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460422+0000","last_change":"2024-03-28T15:01:32.126177+0000","last_active":"2024-03-28T15:01:33.460422+0000","last_peered":"2024-03-28T15:01:33.460422+0000","last_clean":"2024-03-28T15:01:33.460422+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:01:33.460422+0000","last_undegraded":"2024-03-28T15:01:33.460422+0000","last_fullsized":"2024-03-28T15:01:33.460422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126124+0000","last_change":"2024-03-28T15:01:31.094837+0000","last_active":"2024-03-28T15:01:37.126124+0000","last_peered":"2024-03-28T15:01:37.126124+0000","last_clean":"2024-03-28T15:01:37.126124+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:01:37.126124+0000","last_undegraded":"2024-03-28T15:01:37.126124+0000","last_fullsized":"2024-03-28T15:01:37.126124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460675+0000","last_change":"2024-03-28T15:01:32.113803+0000","last_active":"2024-03-28T15:01:33.460675+0000","last_peered":"2024-03-28T15:01:33.460675+0000","last_clean":"2024-03-28T15:01:33.460675+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:01:33.460675+0000","last_undegraded":"2024-03-28T15:01:33.460675+0000","last_fullsized":"2024-03-28T15:01:33.460675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.477351+0000","last_change":"2024-03-28T15:01:31.092455+0000","last_active":"2024-03-28T15:01:34.477351+0000","last_peered":"2024-03-28T15:01:34.477351+0000","last_clean":"2024-03-28T15:01:34.477351+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:01:34.477351+0000","last_undegraded":"2024-03-28T15:01:34.477351+0000","last_fullsized":"2024-03-28T15:01:34.477351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:04.636216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460748+0000","last_change":"2024-03-28T15:01:32.117065+0000","last_active":"2024-03-28T15:01:33.460748+0000","last_peered":"2024-03-28T15:01:33.460748+0000","last_clean":"2024-03-28T15:01:33.460748+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:01:33.460748+0000","last_undegraded":"2024-03-28T15:01:33.460748+0000","last_fullsized":"2024-03-28T15:01:33.460748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:18.895023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.123413+0000","last_change":"2024-03-28T15:01:31.092175+0000","last_active":"2024-03-28T15:01:37.123413+0000","last_peered":"2024-03-28T15:01:37.123413+0000","last_clean":"2024-03-28T15:01:37.123413+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:01:37.123413+0000","last_undegraded":"2024-03-28T15:01:37.123413+0000","last_fullsized":"2024-03-28T15:01:37.123413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461324+0000","last_change":"2024-03-28T15:01:32.114975+0000","last_active":"2024-03-28T15:01:33.461324+0000","last_peered":"2024-03-28T15:01:33.461324+0000","last_clean":"2024-03-28T15:01:33.461324+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:01:33.461324+0000","last_undegraded":"2024-03-28T15:01:33.461324+0000","last_fullsized":"2024-03-28T15:01:33.461324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:18:35.681340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.482725+0000","last_change":"2024-03-28T15:01:31.097901+0000","last_active":"2024-03-28T15:01:34.482725+0000","last_peered":"2024-03-28T15:01:34.482725+0000","last_clean":"2024-03-28T15:01:34.482725+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:01:34.482725+0000","last_undegraded":"2024-03-28T15:01:34.482725+0000","last_fullsized":"2024-03-28T15:01:34.482725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108151+0000","last_change":"2024-03-28T15:01:32.120646+0000","last_active":"2024-03-28T15:01:33.108151+0000","last_peered":"2024-03-28T15:01:33.108151+0000","last_clean":"2024-03-28T15:01:33.108151+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:01:33.108151+0000","last_undegraded":"2024-03-28T15:01:33.108151+0000","last_fullsized":"2024-03-28T15:01:33.108151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:49.705553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460197+0000","last_change":"2024-03-28T15:01:31.086634+0000","last_active":"2024-03-28T15:01:33.460197+0000","last_peered":"2024-03-28T15:01:33.460197+0000","last_clean":"2024-03-28T15:01:33.460197+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:01:33.460197+0000","last_undegraded":"2024-03-28T15:01:33.460197+0000","last_fullsized":"2024-03-28T15:01:33.460197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461104+0000","last_change":"2024-03-28T15:01:32.112493+0000","last_active":"2024-03-28T15:01:33.461104+0000","last_peered":"2024-03-28T15:01:33.461104+0000","last_clean":"2024-03-28T15:01:33.461104+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:01:33.461104+0000","last_undegraded":"2024-03-28T15:01:33.461104+0000","last_fullsized":"2024-03-28T15:01:33.461104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461052+0000","last_change":"2024-03-28T15:01:20.724163+0000","last_active":"2024-03-28T15:01:33.461052+0000","last_peered":"2024-03-28T15:01:33.461052+0000","last_clean":"2024-03-28T15:01:33.461052+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:01:33.461052+0000","last_undegraded":"2024-03-28T15:01:33.461052+0000","last_fullsized":"2024-03-28T15:01:33.461052+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_clean_scrub_stamp":"2024-03-28T15:01:19.548090+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:15.791893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107058+0000","last_change":"2024-03-28T15:01:31.085453+0000","last_active":"2024-03-28T15:01:33.107058+0000","last_peered":"2024-03-28T15:01:33.107058+0000","last_clean":"2024-03-28T15:01:33.107058+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:01:33.107058+0000","last_undegraded":"2024-03-28T15:01:33.107058+0000","last_fullsized":"2024-03-28T15:01:33.107058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460326+0000","last_change":"2024-03-28T15:01:32.118678+0000","last_active":"2024-03-28T15:01:33.460326+0000","last_peered":"2024-03-28T15:01:33.460326+0000","last_clean":"2024-03-28T15:01:33.460326+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:01:33.460326+0000","last_undegraded":"2024-03-28T15:01:33.460326+0000","last_fullsized":"2024-03-28T15:01:33.460326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:55:20.591124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461116+0000","last_change":"2024-03-28T15:01:31.095439+0000","last_active":"2024-03-28T15:01:33.461116+0000","last_peered":"2024-03-28T15:01:33.461116+0000","last_clean":"2024-03-28T15:01:33.461116+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:01:33.461116+0000","last_undegraded":"2024-03-28T15:01:33.461116+0000","last_fullsized":"2024-03-28T15:01:33.461116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460849+0000","last_change":"2024-03-28T15:01:32.122655+0000","last_active":"2024-03-28T15:01:33.460849+0000","last_peered":"2024-03-28T15:01:33.460849+0000","last_clean":"2024-03-28T15:01:33.460849+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:01:33.460849+0000","last_undegraded":"2024-03-28T15:01:33.460849+0000","last_fullsized":"2024-03-28T15:01:33.460849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:05.942532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460385+0000","last_change":"2024-03-28T15:01:31.093265+0000","last_active":"2024-03-28T15:01:33.460385+0000","last_peered":"2024-03-28T15:01:33.460385+0000","last_clean":"2024-03-28T15:01:33.460385+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:01:33.460385+0000","last_undegraded":"2024-03-28T15:01:33.460385+0000","last_fullsized":"2024-03-28T15:01:33.460385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104892+0000","last_change":"2024-03-28T15:01:32.108597+0000","last_active":"2024-03-28T15:01:33.104892+0000","last_peered":"2024-03-28T15:01:33.104892+0000","last_clean":"2024-03-28T15:01:33.104892+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:01:33.104892+0000","last_undegraded":"2024-03-28T15:01:33.104892+0000","last_fullsized":"2024-03-28T15:01:33.104892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108209+0000","last_change":"2024-03-28T15:01:31.094750+0000","last_active":"2024-03-28T15:01:33.108209+0000","last_peered":"2024-03-28T15:01:33.108209+0000","last_clean":"2024-03-28T15:01:33.108209+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:01:33.108209+0000","last_undegraded":"2024-03-28T15:01:33.108209+0000","last_fullsized":"2024-03-28T15:01:33.108209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.106970+0000","last_change":"2024-03-28T15:01:32.121637+0000","last_active":"2024-03-28T15:01:33.106970+0000","last_peered":"2024-03-28T15:01:33.106970+0000","last_clean":"2024-03-28T15:01:33.106970+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:01:33.106970+0000","last_undegraded":"2024-03-28T15:01:33.106970+0000","last_fullsized":"2024-03-28T15:01:33.106970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:24.747778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104874+0000","last_change":"2024-03-28T15:01:31.093706+0000","last_active":"2024-03-28T15:01:33.104874+0000","last_peered":"2024-03-28T15:01:33.104874+0000","last_clean":"2024-03-28T15:01:33.104874+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:01:33.104874+0000","last_undegraded":"2024-03-28T15:01:33.104874+0000","last_fullsized":"2024-03-28T15:01:33.104874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.108112+0000","last_change":"2024-03-28T15:01:32.122840+0000","last_active":"2024-03-28T15:01:33.108112+0000","last_peered":"2024-03-28T15:01:33.108112+0000","last_clean":"2024-03-28T15:01:33.108112+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:01:33.108112+0000","last_undegraded":"2024-03-28T15:01:33.108112+0000","last_fullsized":"2024-03-28T15:01:33.108112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:49.735280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461371+0000","last_change":"2024-03-28T15:01:31.095126+0000","last_active":"2024-03-28T15:01:33.461371+0000","last_peered":"2024-03-28T15:01:33.461371+0000","last_clean":"2024-03-28T15:01:33.461371+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:01:33.461371+0000","last_undegraded":"2024-03-28T15:01:33.461371+0000","last_fullsized":"2024-03-28T15:01:33.461371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:21.669535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104885+0000","last_change":"2024-03-28T15:01:32.111925+0000","last_active":"2024-03-28T15:01:33.104885+0000","last_peered":"2024-03-28T15:01:33.104885+0000","last_clean":"2024-03-28T15:01:33.104885+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:01:33.104885+0000","last_undegraded":"2024-03-28T15:01:33.104885+0000","last_fullsized":"2024-03-28T15:01:33.104885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.482739+0000","last_change":"2024-03-28T15:01:31.093197+0000","last_active":"2024-03-28T15:01:34.482739+0000","last_peered":"2024-03-28T15:01:34.482739+0000","last_clean":"2024-03-28T15:01:34.482739+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:01:34.482739+0000","last_undegraded":"2024-03-28T15:01:34.482739+0000","last_fullsized":"2024-03-28T15:01:34.482739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461153+0000","last_change":"2024-03-28T15:01:32.115432+0000","last_active":"2024-03-28T15:01:33.461153+0000","last_peered":"2024-03-28T15:01:33.461153+0000","last_clean":"2024-03-28T15:01:33.461153+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:01:33.461153+0000","last_undegraded":"2024-03-28T15:01:33.461153+0000","last_fullsized":"2024-03-28T15:01:33.461153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:45.181678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.125948+0000","last_change":"2024-03-28T15:01:31.088204+0000","last_active":"2024-03-28T15:01:37.125948+0000","last_peered":"2024-03-28T15:01:37.125948+0000","last_clean":"2024-03-28T15:01:37.125948+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:01:37.125948+0000","last_undegraded":"2024-03-28T15:01:37.125948+0000","last_fullsized":"2024-03-28T15:01:37.125948+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.461225+0000","last_change":"2024-03-28T15:01:32.114331+0000","last_active":"2024-03-28T15:01:33.461225+0000","last_peered":"2024-03-28T15:01:33.461225+0000","last_clean":"2024-03-28T15:01:33.461225+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:01:33.461225+0000","last_undegraded":"2024-03-28T15:01:33.461225+0000","last_fullsized":"2024-03-28T15:01:33.461225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:58.693387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460969+0000","last_change":"2024-03-28T15:01:31.095371+0000","last_active":"2024-03-28T15:01:33.460969+0000","last_peered":"2024-03-28T15:01:33.460969+0000","last_clean":"2024-03-28T15:01:33.460969+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:01:33.460969+0000","last_undegraded":"2024-03-28T15:01:33.460969+0000","last_fullsized":"2024-03-28T15:01:33.460969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107409+0000","last_change":"2024-03-28T15:01:32.110950+0000","last_active":"2024-03-28T15:01:33.107409+0000","last_peered":"2024-03-28T15:01:33.107409+0000","last_clean":"2024-03-28T15:01:33.107409+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:01:33.107409+0000","last_undegraded":"2024-03-28T15:01:33.107409+0000","last_fullsized":"2024-03-28T15:01:33.107409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:18.997081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.105009+0000","last_change":"2024-03-28T15:01:31.094542+0000","last_active":"2024-03-28T15:01:33.105009+0000","last_peered":"2024-03-28T15:01:33.105009+0000","last_clean":"2024-03-28T15:01:33.105009+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:01:33.105009+0000","last_undegraded":"2024-03-28T15:01:33.105009+0000","last_fullsized":"2024-03-28T15:01:33.105009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460466+0000","last_change":"2024-03-28T15:01:32.112158+0000","last_active":"2024-03-28T15:01:33.460466+0000","last_peered":"2024-03-28T15:01:33.460466+0000","last_clean":"2024-03-28T15:01:33.460466+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:01:33.460466+0000","last_undegraded":"2024-03-28T15:01:33.460466+0000","last_fullsized":"2024-03-28T15:01:33.460466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:43.488120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.104884+0000","last_change":"2024-03-28T15:01:31.094440+0000","last_active":"2024-03-28T15:01:33.104884+0000","last_peered":"2024-03-28T15:01:33.104884+0000","last_clean":"2024-03-28T15:01:33.104884+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:01:33.104884+0000","last_undegraded":"2024-03-28T15:01:33.104884+0000","last_fullsized":"2024-03-28T15:01:33.104884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460758+0000","last_change":"2024-03-28T15:01:32.115192+0000","last_active":"2024-03-28T15:01:33.460758+0000","last_peered":"2024-03-28T15:01:33.460758+0000","last_clean":"2024-03-28T15:01:33.460758+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:01:33.460758+0000","last_undegraded":"2024-03-28T15:01:33.460758+0000","last_fullsized":"2024-03-28T15:01:33.460758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:40:59.083089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.125967+0000","last_change":"2024-03-28T15:01:31.096352+0000","last_active":"2024-03-28T15:01:37.125967+0000","last_peered":"2024-03-28T15:01:37.125967+0000","last_clean":"2024-03-28T15:01:37.125967+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:01:37.125967+0000","last_undegraded":"2024-03-28T15:01:37.125967+0000","last_fullsized":"2024-03-28T15:01:37.125967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460555+0000","last_change":"2024-03-28T15:01:32.124700+0000","last_active":"2024-03-28T15:01:33.460555+0000","last_peered":"2024-03-28T15:01:33.460555+0000","last_clean":"2024-03-28T15:01:33.460555+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:01:33.460555+0000","last_undegraded":"2024-03-28T15:01:33.460555+0000","last_fullsized":"2024-03-28T15:01:33.460555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:59.070224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107999+0000","last_change":"2024-03-28T15:01:32.123638+0000","last_active":"2024-03-28T15:01:33.107999+0000","last_peered":"2024-03-28T15:01:33.107999+0000","last_clean":"2024-03-28T15:01:33.107999+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:01:33.107999+0000","last_undegraded":"2024-03-28T15:01:33.107999+0000","last_fullsized":"2024-03-28T15:01:33.107999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:15.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.121539+0000","last_change":"2024-03-28T15:01:31.094281+0000","last_active":"2024-03-28T15:01:37.121539+0000","last_peered":"2024-03-28T15:01:37.121539+0000","last_clean":"2024-03-28T15:01:37.121539+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:01:37.121539+0000","last_undegraded":"2024-03-28T15:01:37.121539+0000","last_fullsized":"2024-03-28T15:01:37.121539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.107988+0000","last_change":"2024-03-28T15:01:32.112496+0000","last_active":"2024-03-28T15:01:33.107988+0000","last_peered":"2024-03-28T15:01:33.107988+0000","last_clean":"2024-03-28T15:01:33.107988+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:01:33.107988+0000","last_undegraded":"2024-03-28T15:01:33.107988+0000","last_fullsized":"2024-03-28T15:01:33.107988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:54.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126177+0000","last_change":"2024-03-28T15:01:31.093773+0000","last_active":"2024-03-28T15:01:37.126177+0000","last_peered":"2024-03-28T15:01:37.126177+0000","last_clean":"2024-03-28T15:01:37.126177+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:01:37.126177+0000","last_undegraded":"2024-03-28T15:01:37.126177+0000","last_fullsized":"2024-03-28T15:01:37.126177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:54.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460976+0000","last_change":"2024-03-28T15:01:32.117275+0000","last_active":"2024-03-28T15:01:33.460976+0000","last_peered":"2024-03-28T15:01:33.460976+0000","last_clean":"2024-03-28T15:01:33.460976+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:01:33.460976+0000","last_undegraded":"2024-03-28T15:01:33.460976+0000","last_fullsized":"2024-03-28T15:01:33.460976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:55.837280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:34.475554+0000","last_change":"2024-03-28T15:01:31.096725+0000","last_active":"2024-03-28T15:01:34.475554+0000","last_peered":"2024-03-28T15:01:34.475554+0000","last_clean":"2024-03-28T15:01:34.475554+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:01:34.475554+0000","last_undegraded":"2024-03-28T15:01:34.475554+0000","last_fullsized":"2024-03-28T15:01:34.475554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:33.460734+0000","last_change":"2024-03-28T15:01:32.120071+0000","last_active":"2024-03-28T15:01:33.460734+0000","last_peered":"2024-03-28T15:01:33.460734+0000","last_clean":"2024-03-28T15:01:33.460734+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:01:33.460734+0000","last_undegraded":"2024-03-28T15:01:33.460734+0000","last_fullsized":"2024-03-28T15:01:33.460734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:39.572893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T15:01:37.126217+0000","last_change":"2024-03-28T15:01:31.095351+0000","last_active":"2024-03-28T15:01:37.126217+0000","last_peered":"2024-03-28T15:01:37.126217+0000","last_clean":"2024-03-28T15:01:37.126217+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:01:37.126217+0000","last_undegraded":"2024-03-28T15:01:37.126217+0000","last_fullsized":"2024-03-28T15:01:37.126217+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":73802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":73802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:01:51.909 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.046+0000 7f124bfff640 1 Processor -- start 2024-03-28T15:01:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.046+0000 7f124bfff640 1 -- start start 2024-03-28T15:01:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124bfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f124c154920 0x7f124c15b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124bfff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f124c0726d0 0x7f124c0cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124bfff640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f124c0ccbd0 con 0x7f124c0b2f70 2024-03-28T15:01:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124bfff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f124c0cb740 con 0x7f124c154920 2024-03-28T15:01:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124bfff640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f124c0cceb0 con 0x7f124c0726d0 2024-03-28T15:01:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124affd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f124c154920 0x7f124c15b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124affd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f124c154920 0x7f124c15b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42404/0 (socket says 172.21.15.186:42404) 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124a7fc640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f124c0b2f70 0x7f124c0d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:59352/0 (socket says 172.21.15.186:59352) 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124affd640 1 -- 172.21.15.186:0/238169975 learned_addr learned my addr 172.21.15.186:0/238169975 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124b7fe640 1 --2- 172.21.15.186:0/238169975 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f124c0726d0 0x7f124c0cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f1249ffb640 1 -- 172.21.15.186:0/238169975 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 558455028 0 0) 0x7f124c0ccbd0 con 0x7f124c0b2f70 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124affd640 1 -- 172.21.15.186:0/238169975 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f124c0726d0 msgr2=0x7f124c0cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124affd640 1 --2- 172.21.15.186:0/238169975 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f124c0726d0 0x7f124c0cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124affd640 1 -- 172.21.15.186:0/238169975 >> v1:172.21.15.191:6789/0 conn(0x7f124c0b2f70 legacy=0x7f124c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.047+0000 7f124affd640 1 -- 172.21.15.186:0/238169975 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f124c0cd190 con 0x7f124c154920 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.048+0000 7f124affd640 1 --2- 172.21.15.186:0/238169975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f124c154920 0x7f124c15b130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f123c009e80 tx=0x7f123c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727fea5d8de95538 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.048+0000 7f1249ffb640 1 -- 172.21.15.186:0/238169975 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f123c02caf0 con 0x7f124c154920 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.048+0000 7f1249ffb640 1 -- 172.21.15.186:0/238169975 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f123c02cc90 con 0x7f124c154920 2024-03-28T15:01:52.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.048+0000 7f1249ffb640 1 -- 172.21.15.186:0/238169975 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f123c034610 con 0x7f124c154920 2024-03-28T15:01:52.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.049+0000 7f124bfff640 1 -- 172.21.15.186:0/238169975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f124c154920 msgr2=0x7f124c15b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.049+0000 7f124bfff640 1 --2- 172.21.15.186:0/238169975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f124c154920 0x7f124c15b130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f123c009e80 tx=0x7f123c02bf50 comp rx=0 tx=0).stop 2024-03-28T15:01:52.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.049+0000 7f124bfff640 1 -- 172.21.15.186:0/238169975 shutdown_connections 2024-03-28T15:01:52.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.049+0000 7f124bfff640 1 --2- 172.21.15.186:0/238169975 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f124c0726d0 0x7f124c0cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.049+0000 7f124bfff640 1 --2- 172.21.15.186:0/238169975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f124c154920 0x7f124c15b130 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.049+0000 7f124bfff640 1 -- 172.21.15.186:0/238169975 >> 172.21.15.186:0/238169975 conn(0x7f124c0735c0 msgr2=0x7f124c0739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:52.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.049+0000 7f124bfff640 1 -- 172.21.15.186:0/238169975 shutdown_connections 2024-03-28T15:01:52.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.049+0000 7f124bfff640 1 -- 172.21.15.186:0/238169975 wait complete. 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.050+0000 7f124bfff640 1 Processor -- start 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.050+0000 7f124bfff640 1 -- start start 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f124c0726d0 0x7f124c1567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f124c154920 0x7f124c14d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124bfff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f124c14e4b0 0x7f124c1f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124bfff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f124c154d00 con 0x7f124c154920 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124bfff640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f124c0cd750 con 0x7f124c0726d0 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124bfff640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f124c0d0ca0 con 0x7f124c14e4b0 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124a7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f124c154920 0x7f124c14d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124a7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f124c154920 0x7f124c14d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42408/0 (socket says 172.21.15.186:42408) 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124a7fc640 1 -- 172.21.15.186:0/2021754255 learned_addr learned my addr 172.21.15.186:0/2021754255 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124affd640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f124c0726d0 0x7f124c1567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124b7fe640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f124c14e4b0 0x7f124c1f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124a7fc640 1 -- 172.21.15.186:0/2021754255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f124c14e4b0 msgr2=0x7f124c1f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124a7fc640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f124c14e4b0 0x7f124c1f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124a7fc640 1 -- 172.21.15.186:0/2021754255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f124c0726d0 msgr2=0x7f124c1567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124a7fc640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f124c0726d0 0x7f124c1567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124a7fc640 1 -- 172.21.15.186:0/2021754255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f124c156370 con 0x7f124c154920 2024-03-28T15:01:52.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124affd640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f124c0726d0 0x7f124c1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:52.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124a7fc640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f124c154920 0x7f124c14d210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f124000c460 tx=0x7f124000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:52.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f1233fff640 1 -- 172.21.15.186:0/2021754255 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240018070 con 0x7f124c154920 2024-03-28T15:01:52.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f1233fff640 1 -- 172.21.15.186:0/2021754255 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12400074e0 con 0x7f124c154920 2024-03-28T15:01:52.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f1233fff640 1 -- 172.21.15.186:0/2021754255 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240004a50 con 0x7f124c154920 2024-03-28T15:01:52.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124bfff640 1 -- 172.21.15.186:0/2021754255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f124c14b750 con 0x7f124c154920 2024-03-28T15:01:52.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.051+0000 7f124bfff640 1 -- 172.21.15.186:0/2021754255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f124c14ba10 con 0x7f124c154920 2024-03-28T15:01:52.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.052+0000 7f124bfff640 1 -- 172.21.15.186:0/2021754255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f124c069500 con 0x7f124c154920 2024-03-28T15:01:52.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.053+0000 7f1233fff640 1 -- 172.21.15.186:0/2021754255 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f124001b070 con 0x7f124c154920 2024-03-28T15:01:52.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.053+0000 7f1233fff640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f123806c560 0x7f123806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.053+0000 7f124affd640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f123806c560 0x7f123806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.053+0000 7f1233fff640 1 -- 172.21.15.186:0/2021754255 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f124008a890 con 0x7f124c154920 2024-03-28T15:01:52.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.054+0000 7f124affd640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f123806c560 0x7f123806ea20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f123c035000 tx=0x7f123c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:52.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.058+0000 7f1233fff640 1 -- 172.21.15.186:0/2021754255 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f1240058920 con 0x7f124c154920 2024-03-28T15:01:52.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.235+0000 7f124bfff640 1 -- 172.21.15.186:0/2021754255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f124c062ad0 con 0x7f124c154920 2024-03-28T15:01:52.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.237+0000 7f1233fff640 1 -- 172.21.15.186:0/2021754255 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f124005cc20 con 0x7f124c154920 2024-03-28T15:01:52.237 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:52.238 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:33.099254+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:52.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 -- 172.21.15.186:0/2021754255 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f123806c560 msgr2=0x7f123806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f123806c560 0x7f123806ea20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f123c035000 tx=0x7f123c030000 comp rx=0 tx=0).stop 2024-03-28T15:01:52.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 -- 172.21.15.186:0/2021754255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f124c154920 msgr2=0x7f124c14d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f124c154920 0x7f124c14d210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f124000c460 tx=0x7f124000c920 comp rx=0 tx=0).stop 2024-03-28T15:01:52.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 -- 172.21.15.186:0/2021754255 shutdown_connections 2024-03-28T15:01:52.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f123806c560 0x7f123806ea20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f124c14e4b0 0x7f124c1f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f124c154920 0x7f124c14d210 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 --2- 172.21.15.186:0/2021754255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f124c0726d0 0x7f124c1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 -- 172.21.15.186:0/2021754255 >> 172.21.15.186:0/2021754255 conn(0x7f124c0735c0 msgr2=0x7f124c0cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:52.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 -- 172.21.15.186:0/2021754255 shutdown_connections 2024-03-28T15:01:52.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.239+0000 7f124bfff640 1 -- 172.21.15.186:0/2021754255 wait complete. 2024-03-28T15:01:52.251 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:01:18.661274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:01:29.512445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:01:30.388340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:01:52.251 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:01:52.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.355+0000 7f5967246640 1 Processor -- start 2024-03-28T15:01:52.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.355+0000 7f5967246640 1 -- start start 2024-03-28T15:01:52.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.356+0000 7f5967246640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5960071a80 0x7f5960071f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.356+0000 7f5967246640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59600cb6e0 con 0x7f596015a7c0 2024-03-28T15:01:52.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.356+0000 7f5967246640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59600cd620 con 0x7f5960154b60 2024-03-28T15:01:52.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.356+0000 7f5967246640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f5960072820 con 0x7f5960071a80 2024-03-28T15:01:52.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.356+0000 7f59657e3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5960071a80 0x7f5960071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.356+0000 7f595ffff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f596015a7c0 0x7f59600ccf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49804/0 (socket says 172.21.15.186:49804) 2024-03-28T15:01:52.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.356+0000 7f595ffff640 1 -- 172.21.15.186:0/1681383447 learned_addr learned my addr 172.21.15.186:0/1681383447 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:52.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.356+0000 7f5964fe2640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f5960154b60 0x7f59600cafd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:59354/0 (socket says 172.21.15.186:59354) 2024-03-28T15:01:52.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.357+0000 7f595f7fe640 1 -- 172.21.15.186:0/1681383447 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 692053971 0 0) 0x7f5950001040 con 0x7f5960154b60 2024-03-28T15:01:52.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.357+0000 7f59657e3640 1 -- 172.21.15.186:0/1681383447 >> v1:172.21.15.191:6789/0 conn(0x7f5960154b60 legacy=0x7f59600cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.357+0000 7f59657e3640 1 -- 172.21.15.186:0/1681383447 >> v1:172.21.15.186:6789/0 conn(0x7f596015a7c0 legacy=0x7f59600ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.357+0000 7f59657e3640 1 -- 172.21.15.186:0/1681383447 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f596015b760 con 0x7f5960071a80 2024-03-28T15:01:52.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.357+0000 7f59657e3640 1 --2- 172.21.15.186:0/1681383447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5960071a80 0x7f5960071f10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5958009d90 tx=0x7f595802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab2d2ca79926153 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:52.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.358+0000 7f595f7fe640 1 -- 172.21.15.186:0/1681383447 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5958039070 con 0x7f5960071a80 2024-03-28T15:01:52.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.358+0000 7f595f7fe640 1 -- 172.21.15.186:0/1681383447 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f595802cb40 con 0x7f5960071a80 2024-03-28T15:01:52.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.358+0000 7f595f7fe640 1 -- 172.21.15.186:0/1681383447 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5958034440 con 0x7f5960071a80 2024-03-28T15:01:52.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.358+0000 7f5967246640 1 -- 172.21.15.186:0/1681383447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5960071a80 msgr2=0x7f5960071f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.358+0000 7f5967246640 1 --2- 172.21.15.186:0/1681383447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5960071a80 0x7f5960071f10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5958009d90 tx=0x7f595802be10 comp rx=0 tx=0).stop 2024-03-28T15:01:52.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.358+0000 7f5967246640 1 -- 172.21.15.186:0/1681383447 shutdown_connections 2024-03-28T15:01:52.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.358+0000 7f5967246640 1 --2- 172.21.15.186:0/1681383447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5960071a80 0x7f5960071f10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.358+0000 7f5967246640 1 -- 172.21.15.186:0/1681383447 >> 172.21.15.186:0/1681383447 conn(0x7f5960073b80 msgr2=0x7f596006b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.358+0000 7f5967246640 1 -- 172.21.15.186:0/1681383447 shutdown_connections 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.358+0000 7f5967246640 1 -- 172.21.15.186:0/1681383447 wait complete. 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.359+0000 7f5967246640 1 Processor -- start 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.359+0000 7f5967246640 1 -- start start 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f5967246640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5960071a80 0x7f596014b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f5967246640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5960154b60 0x7f59601e4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f5967246640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f596015a7c0 0x7f59601449a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f5967246640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f596015b480 con 0x7f5960071a80 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f5967246640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f596015b1a0 con 0x7f5960154b60 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f5967246640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f59600cc710 con 0x7f596015a7c0 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f5964fe2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5960071a80 0x7f596014b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f5964fe2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5960071a80 0x7f596014b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42430/0 (socket says 172.21.15.186:42430) 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f5964fe2640 1 -- 172.21.15.186:0/3538574920 learned_addr learned my addr 172.21.15.186:0/3538574920 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f595ffff640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5960154b60 0x7f59601e4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f59657e3640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f596015a7c0 0x7f59601449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f5964fe2640 1 -- 172.21.15.186:0/3538574920 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f596015a7c0 msgr2=0x7f59601449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.360+0000 7f5964fe2640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f596015a7c0 0x7f59601449a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.361+0000 7f5964fe2640 1 -- 172.21.15.186:0/3538574920 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5960154b60 msgr2=0x7f59601e4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.361+0000 7f5964fe2640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5960154b60 0x7f59601e4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.361+0000 7f5964fe2640 1 -- 172.21.15.186:0/3538574920 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5960142530 con 0x7f5960071a80 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.361+0000 7f5964fe2640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5960071a80 0x7f596014b9f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f595000c520 tx=0x7f595000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.361+0000 7f595d7fa640 1 -- 172.21.15.186:0/3538574920 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950018070 con 0x7f5960071a80 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.361+0000 7f595d7fa640 1 -- 172.21.15.186:0/3538574920 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59500074e0 con 0x7f5960071a80 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.361+0000 7f5967246640 1 -- 172.21.15.186:0/3538574920 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f596014d700 con 0x7f5960071a80 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.361+0000 7f595d7fa640 1 -- 172.21.15.186:0/3538574920 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950004a20 con 0x7f5960071a80 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.361+0000 7f5967246640 1 -- 172.21.15.186:0/3538574920 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59601e6900 con 0x7f5960071a80 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.362+0000 7f5967246640 1 -- 172.21.15.186:0/3538574920 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59600686d0 con 0x7f5960071a80 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.362+0000 7f595d7fa640 1 -- 172.21.15.186:0/3538574920 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f595001b070 con 0x7f5960071a80 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.363+0000 7f595d7fa640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f594c06c530 0x7f594c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.363+0000 7f595d7fa640 1 -- 172.21.15.186:0/3538574920 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f595008a850 con 0x7f5960071a80 2024-03-28T15:01:52.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.363+0000 7f595ffff640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f594c06c530 0x7f594c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.363+0000 7f595ffff640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f594c06c530 0x7f594c06e9f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f596014b120 tx=0x7f5954009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:52.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.367+0000 7f595d7fa640 1 -- 172.21.15.186:0/3538574920 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f59600cd620 con 0x7f5960071a80 2024-03-28T15:01:52.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.541+0000 7f5967246640 1 -- 172.21.15.186:0/3538574920 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f59600c6a20 con 0x7f5960071a80 2024-03-28T15:01:52.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.541+0000 7f595d7fa640 1 -- 172.21.15.186:0/3538574920 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f59500588b0 con 0x7f5960071a80 2024-03-28T15:01:52.542 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 1 2024-03-28T15:01:52.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.543+0000 7f5967246640 1 -- 172.21.15.186:0/3538574920 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f594c06c530 msgr2=0x7f594c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.543+0000 7f5967246640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f594c06c530 0x7f594c06e9f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f596014b120 tx=0x7f5954009040 comp rx=0 tx=0).stop 2024-03-28T15:01:52.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.543+0000 7f5967246640 1 -- 172.21.15.186:0/3538574920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5960071a80 msgr2=0x7f596014b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.543+0000 7f5967246640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5960071a80 0x7f596014b9f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f595000c520 tx=0x7f595000c9e0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.543+0000 7f5967246640 1 -- 172.21.15.186:0/3538574920 shutdown_connections 2024-03-28T15:01:52.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.543+0000 7f5967246640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f594c06c530 0x7f594c06e9f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.543+0000 7f5967246640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f596015a7c0 0x7f59601449a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.543+0000 7f5967246640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5960154b60 0x7f59601e4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.543+0000 7f5967246640 1 --2- 172.21.15.186:0/3538574920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5960071a80 0x7f596014b9f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.544+0000 7f5967246640 1 -- 172.21.15.186:0/3538574920 >> 172.21.15.186:0/3538574920 conn(0x7f5960073b80 msgr2=0x7f59601590d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:52.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.544+0000 7f5967246640 1 -- 172.21.15.186:0/3538574920 shutdown_connections 2024-03-28T15:01:52.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.544+0000 7f5967246640 1 -- 172.21.15.186:0/3538574920 wait complete. 2024-03-28T15:01:52.652 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:01:52.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.756+0000 7f4829965640 1 Processor -- start 2024-03-28T15:01:52.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.756+0000 7f4829965640 1 -- start start 2024-03-28T15:01:52.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.757+0000 7f4829965640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f482415a7c0 0x7f48240ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.757+0000 7f4829965640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48240cb6e0 con 0x7f4824154b60 2024-03-28T15:01:52.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.757+0000 7f4829965640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f482415b760 con 0x7f4824071a80 2024-03-28T15:01:52.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.757+0000 7f4829965640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4824072820 con 0x7f482415a7c0 2024-03-28T15:01:52.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.757+0000 7f4822ffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4824154b60 0x7f48240cafd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49820/0 (socket says 172.21.15.186:49820) 2024-03-28T15:01:52.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.757+0000 7f4822ffd640 1 -- 172.21.15.186:0/831268090 learned_addr learned my addr 172.21.15.186:0/831268090 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:01:52.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.757+0000 7f48227fc640 1 --2- 172.21.15.186:0/831268090 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f482415a7c0 0x7f48240ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.757+0000 7f4821ffb640 1 -- 172.21.15.186:0/831268090 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 304635267 0 0) 0x7f48240cb6e0 con 0x7f4824154b60 2024-03-28T15:01:52.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.757+0000 7f48227fc640 1 -- 172.21.15.186:0/831268090 >> v1:172.21.15.191:6789/0 conn(0x7f4824071a80 legacy=0x7f4824165140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.757+0000 7f48227fc640 1 -- 172.21.15.186:0/831268090 >> v1:172.21.15.186:6789/0 conn(0x7f4824154b60 legacy=0x7f48240cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.757+0000 7f48227fc640 1 -- 172.21.15.186:0/831268090 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482415aec0 con 0x7f482415a7c0 2024-03-28T15:01:52.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.758+0000 7f48227fc640 1 --2- 172.21.15.186:0/831268090 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f482415a7c0 0x7f48240ccf10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4818009750 tx=0x7f481802b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0036f4de2f67c68 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:52.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.758+0000 7f4821ffb640 1 -- 172.21.15.186:0/831268090 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4818038070 con 0x7f482415a7c0 2024-03-28T15:01:52.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.758+0000 7f4821ffb640 1 -- 172.21.15.186:0/831268090 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4818031710 con 0x7f482415a7c0 2024-03-28T15:01:52.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.758+0000 7f4821ffb640 1 -- 172.21.15.186:0/831268090 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4818031ae0 con 0x7f482415a7c0 2024-03-28T15:01:52.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.759+0000 7f4829965640 1 -- 172.21.15.186:0/831268090 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f482415a7c0 msgr2=0x7f48240ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.759+0000 7f4829965640 1 --2- 172.21.15.186:0/831268090 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f482415a7c0 0x7f48240ccf10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4818009750 tx=0x7f481802b7b0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.759+0000 7f4829965640 1 -- 172.21.15.186:0/831268090 shutdown_connections 2024-03-28T15:01:52.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.759+0000 7f4829965640 1 --2- 172.21.15.186:0/831268090 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f482415a7c0 0x7f48240ccf10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.759+0000 7f4829965640 1 -- 172.21.15.186:0/831268090 >> 172.21.15.186:0/831268090 conn(0x7f4824073b80 msgr2=0x7f482406b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:52.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.759+0000 7f4829965640 1 -- 172.21.15.186:0/831268090 shutdown_connections 2024-03-28T15:01:52.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.759+0000 7f4829965640 1 -- 172.21.15.186:0/831268090 wait complete. 2024-03-28T15:01:52.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.760+0000 7f4829965640 1 Processor -- start 2024-03-28T15:01:52.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.760+0000 7f4829965640 1 -- start start 2024-03-28T15:01:52.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4829965640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4824071a80 0x7f482415e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4829965640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4824154b60 0x7f482415c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4829965640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f482415a7c0 0x7f48240c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4829965640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48240cc710 con 0x7f4824071a80 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4829965640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f482415b480 con 0x7f482415a7c0 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4829965640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f482415abe0 con 0x7f4824154b60 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4822ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4824071a80 0x7f482415e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4822ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4824071a80 0x7f482415e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42442/0 (socket says 172.21.15.186:42442) 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4822ffd640 1 -- 172.21.15.186:0/2618415134 learned_addr learned my addr 172.21.15.186:0/2618415134 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f48227fc640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4824154b60 0x7f482415c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f48237fe640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f482415a7c0 0x7f48240c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4822ffd640 1 -- 172.21.15.186:0/2618415134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4824154b60 msgr2=0x7f482415c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4822ffd640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4824154b60 0x7f482415c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4822ffd640 1 -- 172.21.15.186:0/2618415134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f482415a7c0 msgr2=0x7f48240c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4822ffd640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f482415a7c0 0x7f48240c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f4822ffd640 1 -- 172.21.15.186:0/2618415134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48240c8680 con 0x7f4824071a80 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.761+0000 7f48227fc640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4824154b60 0x7f482415c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.762+0000 7f4822ffd640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4824071a80 0x7f482415e000 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f481400c8f0 tx=0x7f481400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.762+0000 7f480bfff640 1 -- 172.21.15.186:0/2618415134 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4814018070 con 0x7f4824071a80 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.762+0000 7f480bfff640 1 -- 172.21.15.186:0/2618415134 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f481400a640 con 0x7f4824071a80 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.762+0000 7f4829965640 1 -- 172.21.15.186:0/2618415134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48240d0100 con 0x7f4824071a80 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.762+0000 7f480bfff640 1 -- 172.21.15.186:0/2618415134 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481400aa30 con 0x7f4824071a80 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.762+0000 7f4829965640 1 -- 172.21.15.186:0/2618415134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48240d06c0 con 0x7f4824071a80 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.763+0000 7f4829965640 1 -- 172.21.15.186:0/2618415134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48240686d0 con 0x7f4824071a80 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.763+0000 7f480bfff640 1 -- 172.21.15.186:0/2618415134 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f481401b070 con 0x7f4824071a80 2024-03-28T15:01:52.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.764+0000 7f480bfff640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f481006c530 0x7f481006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:52.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.764+0000 7f48227fc640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f481006c530 0x7f481006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:52.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.764+0000 7f480bfff640 1 -- 172.21.15.186:0/2618415134 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f481408abd0 con 0x7f4824071a80 2024-03-28T15:01:52.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.764+0000 7f48227fc640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f481006c530 0x7f481006e9f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4818035000 tx=0x7f4818008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:52.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.768+0000 7f480bfff640 1 -- 172.21.15.186:0/2618415134 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4814058c30 con 0x7f4824071a80 2024-03-28T15:01:52.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.944+0000 7f4829965640 1 -- 172.21.15.186:0/2618415134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f48240cc0d0 con 0x7f4824071a80 2024-03-28T15:01:52.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.944+0000 7f480bfff640 1 -- 172.21.15.186:0/2618415134 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f481405cf30 con 0x7f4824071a80 2024-03-28T15:01:52.945 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 32 2024-03-28T15:01:52.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 -- 172.21.15.186:0/2618415134 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f481006c530 msgr2=0x7f481006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f481006c530 0x7f481006e9f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4818035000 tx=0x7f4818008030 comp rx=0 tx=0).stop 2024-03-28T15:01:52.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 -- 172.21.15.186:0/2618415134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4824071a80 msgr2=0x7f482415e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:52.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4824071a80 0x7f482415e000 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f481400c8f0 tx=0x7f481400cdb0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 -- 172.21.15.186:0/2618415134 shutdown_connections 2024-03-28T15:01:52.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f481006c530 0x7f481006e9f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f482415a7c0 0x7f48240c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4824154b60 0x7f482415c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 --2- 172.21.15.186:0/2618415134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4824071a80 0x7f482415e000 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:52.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 -- 172.21.15.186:0/2618415134 >> 172.21.15.186:0/2618415134 conn(0x7f4824073b80 msgr2=0x7f48240cb040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:52.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 -- 172.21.15.186:0/2618415134 shutdown_connections 2024-03-28T15:01:52.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:52.946+0000 7f4829965640 1 -- 172.21.15.186:0/2618415134 wait complete. 2024-03-28T15:01:52.958 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:01:53.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.067+0000 7feb827b6640 1 Processor -- start 2024-03-28T15:01:53.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.067+0000 7feb827b6640 1 -- start start 2024-03-28T15:01:53.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb827b6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7feb7c154c40 0x7feb7c0c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb827b6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7feb7c0701e0 0x7feb7c0c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb827b6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb7c0cc5f0 0x7feb7c0cca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb827b6640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7feb7c155020 con 0x7feb7c0701e0 2024-03-28T15:01:53.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb827b6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7feb7c15e860 con 0x7feb7c154c40 2024-03-28T15:01:53.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb827b6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7feb7c15ed90 con 0x7feb7c0cc5f0 2024-03-28T15:01:53.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb7b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7feb7c0701e0 0x7feb7c0c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb7b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7feb7c0701e0 0x7feb7c0c78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42458/0 (socket says 172.21.15.186:42458) 2024-03-28T15:01:53.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb80d53640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb7c0cc5f0 0x7feb7c0cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb7b7fe640 1 -- 172.21.15.186:0/1916285591 learned_addr learned my addr 172.21.15.186:0/1916285591 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb7bfff640 1 --2- 172.21.15.186:0/1916285591 >> v2:172.21.15.191:3300/0 conn(0x7feb7c154c40 0x7feb7c0c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb7b7fe640 1 -- 172.21.15.186:0/1916285591 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb7c0cc5f0 msgr2=0x7feb7c0cca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb7b7fe640 1 --2- 172.21.15.186:0/1916285591 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb7c0cc5f0 0x7feb7c0cca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb7b7fe640 1 -- 172.21.15.186:0/1916285591 >> v2:172.21.15.191:3300/0 conn(0x7feb7c154c40 msgr2=0x7feb7c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb7b7fe640 1 --2- 172.21.15.186:0/1916285591 >> v2:172.21.15.191:3300/0 conn(0x7feb7c154c40 0x7feb7c0c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb7b7fe640 1 -- 172.21.15.186:0/1916285591 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7c15a860 con 0x7feb7c0701e0 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.068+0000 7feb80d53640 1 --2- 172.21.15.186:0/1916285591 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb7c0cc5f0 0x7feb7c0cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.069+0000 7feb7b7fe640 1 --2- 172.21.15.186:0/1916285591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb7c0701e0 0x7feb7c0c78b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0cd320 tx=0x7feb7002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e810f47e4206e699 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.069+0000 7feb7affd640 1 -- 172.21.15.186:0/1916285591 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7002e070 con 0x7feb7c0701e0 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.069+0000 7feb7affd640 1 -- 172.21.15.186:0/1916285591 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb70037070 con 0x7feb7c0701e0 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.069+0000 7feb7affd640 1 -- 172.21.15.186:0/1916285591 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb70031a00 con 0x7feb7c0701e0 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.070+0000 7feb827b6640 1 -- 172.21.15.186:0/1916285591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb7c0701e0 msgr2=0x7feb7c0c78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.070+0000 7feb827b6640 1 --2- 172.21.15.186:0/1916285591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb7c0701e0 0x7feb7c0c78b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0cd320 tx=0x7feb7002b880 comp rx=0 tx=0).stop 2024-03-28T15:01:53.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.070+0000 7feb827b6640 1 -- 172.21.15.186:0/1916285591 shutdown_connections 2024-03-28T15:01:53.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.070+0000 7feb827b6640 1 --2- 172.21.15.186:0/1916285591 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb7c0cc5f0 0x7feb7c0cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.070+0000 7feb827b6640 1 --2- 172.21.15.186:0/1916285591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb7c0701e0 0x7feb7c0c78b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.070+0000 7feb827b6640 1 --2- 172.21.15.186:0/1916285591 >> v2:172.21.15.191:3300/0 conn(0x7feb7c154c40 0x7feb7c0c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.070+0000 7feb827b6640 1 -- 172.21.15.186:0/1916285591 >> 172.21.15.186:0/1916285591 conn(0x7feb7c06df10 msgr2=0x7feb7c06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:53.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.070+0000 7feb827b6640 1 -- 172.21.15.186:0/1916285591 shutdown_connections 2024-03-28T15:01:53.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.070+0000 7feb827b6640 1 -- 172.21.15.186:0/1916285591 wait complete. 2024-03-28T15:01:53.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.071+0000 7feb827b6640 1 Processor -- start 2024-03-28T15:01:53.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.071+0000 7feb827b6640 1 -- start start 2024-03-28T15:01:53.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb827b6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb7c0701e0 0x7feb7c1f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb827b6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb7c0cc5f0 0x7feb7c0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb827b6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb7c154c40 0x7feb7c152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb827b6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c0c7470 con 0x7feb7c0701e0 2024-03-28T15:01:53.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb827b6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c0c7190 con 0x7feb7c154c40 2024-03-28T15:01:53.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb827b6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7feb7c15ad60 con 0x7feb7c0cc5f0 2024-03-28T15:01:53.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb7bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb7c0701e0 0x7feb7c1f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb7bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb7c0701e0 0x7feb7c1f0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42466/0 (socket says 172.21.15.186:42466) 2024-03-28T15:01:53.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb7bfff640 1 -- 172.21.15.186:0/152846801 learned_addr learned my addr 172.21.15.186:0/152846801 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:53.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb7b7fe640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb7c0cc5f0 0x7feb7c0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb80d53640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb7c154c40 0x7feb7c152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb7bfff640 1 -- 172.21.15.186:0/152846801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb7c0cc5f0 msgr2=0x7feb7c0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb7bfff640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb7c0cc5f0 0x7feb7c0c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb7bfff640 1 -- 172.21.15.186:0/152846801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb7c154c40 msgr2=0x7feb7c152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb7bfff640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb7c154c40 0x7feb7c152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb7bfff640 1 -- 172.21.15.186:0/152846801 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7c15bac0 con 0x7feb7c0701e0 2024-03-28T15:01:53.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb7bfff640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb7c0701e0 0x7feb7c1f0bc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00e870 tx=0x7feb6c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:53.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb78ff9640 1 -- 172.21.15.186:0/152846801 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c017070 con 0x7feb7c0701e0 2024-03-28T15:01:53.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb78ff9640 1 -- 172.21.15.186:0/152846801 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb6c00ee60 con 0x7feb7c0701e0 2024-03-28T15:01:53.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb827b6640 1 -- 172.21.15.186:0/152846801 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7c15bd60 con 0x7feb7c0701e0 2024-03-28T15:01:53.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb78ff9640 1 -- 172.21.15.186:0/152846801 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c00cb80 con 0x7feb7c0701e0 2024-03-28T15:01:53.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.072+0000 7feb827b6640 1 -- 172.21.15.186:0/152846801 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb7c15c320 con 0x7feb7c0701e0 2024-03-28T15:01:53.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.073+0000 7feb827b6640 1 -- 172.21.15.186:0/152846801 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb7c069570 con 0x7feb7c0701e0 2024-03-28T15:01:53.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.074+0000 7feb78ff9640 1 -- 172.21.15.186:0/152846801 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7feb6c01a070 con 0x7feb7c0701e0 2024-03-28T15:01:53.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.074+0000 7feb78ff9640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7feb6806c530 0x7feb6806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.074+0000 7feb78ff9640 1 -- 172.21.15.186:0/152846801 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7feb6c08b700 con 0x7feb7c0701e0 2024-03-28T15:01:53.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.075+0000 7feb7b7fe640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7feb6806c530 0x7feb6806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.075+0000 7feb7b7fe640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7feb6806c530 0x7feb6806e9f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feb70036000 tx=0x7feb7003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:53.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.079+0000 7feb78ff9640 1 -- 172.21.15.186:0/152846801 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7feb6c0596d0 con 0x7feb7c0701e0 2024-03-28T15:01:53.257 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.256+0000 7feb827b6640 1 -- 172.21.15.186:0/152846801 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7feb7c071f90 con 0x7feb7c0701e0 2024-03-28T15:01:53.257 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.256+0000 7feb78ff9640 1 -- 172.21.15.186:0/152846801 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7feb6c05d9d0 con 0x7feb7c0701e0 2024-03-28T15:01:53.257 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 64 2024-03-28T15:01:53.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 -- 172.21.15.186:0/152846801 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7feb6806c530 msgr2=0x7feb6806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7feb6806c530 0x7feb6806e9f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feb70036000 tx=0x7feb7003d040 comp rx=0 tx=0).stop 2024-03-28T15:01:53.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 -- 172.21.15.186:0/152846801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb7c0701e0 msgr2=0x7feb7c1f0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb7c0701e0 0x7feb7c1f0bc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00e870 tx=0x7feb6c00ec70 comp rx=0 tx=0).stop 2024-03-28T15:01:53.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 -- 172.21.15.186:0/152846801 shutdown_connections 2024-03-28T15:01:53.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7feb6806c530 0x7feb6806e9f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb7c154c40 0x7feb7c152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb7c0cc5f0 0x7feb7c0c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 --2- 172.21.15.186:0/152846801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb7c0701e0 0x7feb7c1f0bc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 -- 172.21.15.186:0/152846801 >> 172.21.15.186:0/152846801 conn(0x7feb7c06df10 msgr2=0x7feb7c0c90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:53.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 -- 172.21.15.186:0/152846801 shutdown_connections 2024-03-28T15:01:53.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.258+0000 7feb827b6640 1 -- 172.21.15.186:0/152846801 wait complete. 2024-03-28T15:01:53.270 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:01:53.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.376+0000 7f22a8dc7640 1 Processor -- start 2024-03-28T15:01:53.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.376+0000 7f22a8dc7640 1 -- start start 2024-03-28T15:01:53.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a8dc7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f22a4152810 0x7f22a4158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a8dc7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22a4071340 0x7f22a406c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a8dc7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a4072df0 con 0x7f22a406ce60 2024-03-28T15:01:53.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a8dc7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f22a4159500 con 0x7f22a4152810 2024-03-28T15:01:53.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a8dc7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f22a40c8660 con 0x7f22a4071340 2024-03-28T15:01:53.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a259c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f22a4152810 0x7f22a4158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a259c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f22a4152810 0x7f22a4158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42476/0 (socket says 172.21.15.186:42476) 2024-03-28T15:01:53.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a259c640 1 -- 172.21.15.186:0/2480955969 learned_addr learned my addr 172.21.15.186:0/2480955969 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:53.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a1d9b640 1 --2- 172.21.15.186:0/2480955969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22a4071340 0x7f22a406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a259c640 1 -- 172.21.15.186:0/2480955969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22a4071340 msgr2=0x7f22a406c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:53.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a259c640 1 --2- 172.21.15.186:0/2480955969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22a4071340 0x7f22a406c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a259c640 1 -- 172.21.15.186:0/2480955969 >> v1:172.21.15.191:6789/0 conn(0x7f22a406ce60 legacy=0x7f22a40c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.377+0000 7f22a259c640 1 -- 172.21.15.186:0/2480955969 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a4071d50 con 0x7f22a4152810 2024-03-28T15:01:53.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.378+0000 7f22a259c640 1 --2- 172.21.15.186:0/2480955969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a4152810 0x7f22a4158fc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2294009870 tx=0x7f229402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0e2671edc6a5a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:53.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.378+0000 7f22a159a640 1 -- 172.21.15.186:0/2480955969 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294034070 con 0x7f22a4152810 2024-03-28T15:01:53.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.378+0000 7f22a159a640 1 -- 172.21.15.186:0/2480955969 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2294030840 con 0x7f22a4152810 2024-03-28T15:01:53.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.378+0000 7f22a159a640 1 -- 172.21.15.186:0/2480955969 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294030c10 con 0x7f22a4152810 2024-03-28T15:01:53.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.379+0000 7f22a8dc7640 1 -- 172.21.15.186:0/2480955969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a4152810 msgr2=0x7f22a4158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.379+0000 7f22a8dc7640 1 --2- 172.21.15.186:0/2480955969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a4152810 0x7f22a4158fc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2294009870 tx=0x7f229402b980 comp rx=0 tx=0).stop 2024-03-28T15:01:53.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.379+0000 7f22a8dc7640 1 -- 172.21.15.186:0/2480955969 shutdown_connections 2024-03-28T15:01:53.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.379+0000 7f22a8dc7640 1 --2- 172.21.15.186:0/2480955969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22a4071340 0x7f22a406c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.379+0000 7f22a8dc7640 1 --2- 172.21.15.186:0/2480955969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a4152810 0x7f22a4158fc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.379+0000 7f22a8dc7640 1 -- 172.21.15.186:0/2480955969 >> 172.21.15.186:0/2480955969 conn(0x7f22a40c8a20 msgr2=0x7f22a40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:53.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.379+0000 7f22a8dc7640 1 -- 172.21.15.186:0/2480955969 shutdown_connections 2024-03-28T15:01:53.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.379+0000 7f22a8dc7640 1 -- 172.21.15.186:0/2480955969 wait complete. 2024-03-28T15:01:53.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.380+0000 7f22a8dc7640 1 Processor -- start 2024-03-28T15:01:53.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.380+0000 7f22a8dc7640 1 -- start start 2024-03-28T15:01:53.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a8dc7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22a406ce60 0x7f22a4155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a8dc7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22a4071340 0x7f22a41e3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a8dc7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a4152810 0x7f22a4144a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a8dc7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22a40725f0 con 0x7f22a4152810 2024-03-28T15:01:53.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a8dc7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f22a4072310 con 0x7f22a406ce60 2024-03-28T15:01:53.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a8dc7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f22a40728d0 con 0x7f22a4071340 2024-03-28T15:01:53.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a4152810 0x7f22a4144a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a4152810 0x7f22a4144a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42490/0 (socket says 172.21.15.186:42490) 2024-03-28T15:01:53.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a2d9d640 1 -- 172.21.15.186:0/88304323 learned_addr learned my addr 172.21.15.186:0/88304323 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:53.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a259c640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22a406ce60 0x7f22a4155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a1d9b640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22a4071340 0x7f22a41e3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a2d9d640 1 -- 172.21.15.186:0/88304323 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22a4071340 msgr2=0x7f22a41e3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a2d9d640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22a4071340 0x7f22a41e3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a2d9d640 1 -- 172.21.15.186:0/88304323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22a406ce60 msgr2=0x7f22a4155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a2d9d640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22a406ce60 0x7f22a4155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a2d9d640 1 -- 172.21.15.186:0/88304323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a40c2d20 con 0x7f22a4152810 2024-03-28T15:01:53.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a2d9d640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a4152810 0x7f22a4144a10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f229c00c6b0 tx=0x7f229c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:53.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f2292ffd640 1 -- 172.21.15.186:0/88304323 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c003b70 con 0x7f22a4152810 2024-03-28T15:01:53.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f2292ffd640 1 -- 172.21.15.186:0/88304323 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f229c003d10 con 0x7f22a4152810 2024-03-28T15:01:53.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f22a8dc7640 1 -- 172.21.15.186:0/88304323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a40c4480 con 0x7f22a4152810 2024-03-28T15:01:53.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.381+0000 7f2292ffd640 1 -- 172.21.15.186:0/88304323 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c005310 con 0x7f22a4152810 2024-03-28T15:01:53.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.382+0000 7f22a8dc7640 1 -- 172.21.15.186:0/88304323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a40c4a40 con 0x7f22a4152810 2024-03-28T15:01:53.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.383+0000 7f22a8dc7640 1 -- 172.21.15.186:0/88304323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a4068a50 con 0x7f22a4152810 2024-03-28T15:01:53.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.383+0000 7f2292ffd640 1 -- 172.21.15.186:0/88304323 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f229c0054b0 con 0x7f22a4152810 2024-03-28T15:01:53.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.383+0000 7f2292ffd640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f228406c530 0x7f228406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.383+0000 7f2292ffd640 1 -- 172.21.15.186:0/88304323 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f229c08b390 con 0x7f22a4152810 2024-03-28T15:01:53.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.383+0000 7f22a259c640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f228406c530 0x7f228406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.384+0000 7f22a259c640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f228406c530 0x7f228406e9f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2294036000 tx=0x7f2294008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:53.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.388+0000 7f2292ffd640 1 -- 172.21.15.186:0/88304323 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f22a4068a50 con 0x7f22a4152810 2024-03-28T15:01:53.600 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.600+0000 7f22a8dc7640 1 -- 172.21.15.186:0/88304323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22a40c6610 con 0x7f22a4152810 2024-03-28T15:01:53.600 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.600+0000 7f2292ffd640 1 -- 172.21.15.186:0/88304323 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7f22a40c6610 con 0x7f22a4152810 2024-03-28T15:01:53.601 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:53.601 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:01:53.601 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 -- 172.21.15.186:0/88304323 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f228406c530 msgr2=0x7f228406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f228406c530 0x7f228406e9f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2294036000 tx=0x7f2294008030 comp rx=0 tx=0).stop 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 -- 172.21.15.186:0/88304323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a4152810 msgr2=0x7f22a4144a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a4152810 0x7f22a4144a10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f229c00c6b0 tx=0x7f229c00cb70 comp rx=0 tx=0).stop 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 -- 172.21.15.186:0/88304323 shutdown_connections 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f228406c530 0x7f228406e9f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a4152810 0x7f22a4144a10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22a4071340 0x7f22a41e3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 --2- 172.21.15.186:0/88304323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22a406ce60 0x7f22a4155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 -- 172.21.15.186:0/88304323 >> 172.21.15.186:0/88304323 conn(0x7f22a40c8a20 msgr2=0x7f22a40c3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 -- 172.21.15.186:0/88304323 shutdown_connections 2024-03-28T15:01:53.603 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.602+0000 7f22a8dc7640 1 -- 172.21.15.186:0/88304323 wait complete. 2024-03-28T15:01:53.614 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:01:53.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.716+0000 7f42669e2640 1 Processor -- start 2024-03-28T15:01:53.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.716+0000 7f42669e2640 1 -- start start 2024-03-28T15:01:53.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f42669e2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4260152810 0x7f4260158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f42669e2640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4260071340 0x7f426006c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f42669e2640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f426006ce60 0x7f42600c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f42669e2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f4260159500 con 0x7f4260152810 2024-03-28T15:01:53.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f42669e2640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f4260072df0 con 0x7f426006ce60 2024-03-28T15:01:53.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f42669e2640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f42600c8660 con 0x7f4260071340 2024-03-28T15:01:53.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f425ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4260152810 0x7f4260158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f425ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4260152810 0x7f4260158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42498/0 (socket says 172.21.15.186:42498) 2024-03-28T15:01:53.721 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f425ffff640 1 -- 172.21.15.186:0/387345566 learned_addr learned my addr 172.21.15.186:0/387345566 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:53.721 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f4264f7f640 1 --2- 172.21.15.186:0/387345566 >> v2:172.21.15.191:3300/0 conn(0x7f426006ce60 0x7f42600c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.721 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f425f7fe640 1 --2- 172.21.15.186:0/387345566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4260071340 0x7f426006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.721 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f425ffff640 1 -- 172.21.15.186:0/387345566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4260071340 msgr2=0x7f426006c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:53.721 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f425ffff640 1 --2- 172.21.15.186:0/387345566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4260071340 0x7f426006c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f425ffff640 1 -- 172.21.15.186:0/387345566 >> v2:172.21.15.191:3300/0 conn(0x7f426006ce60 msgr2=0x7f42600c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:53.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f425ffff640 1 --2- 172.21.15.186:0/387345566 >> v2:172.21.15.191:3300/0 conn(0x7f426006ce60 0x7f42600c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f425ffff640 1 -- 172.21.15.186:0/387345566 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4260071d50 con 0x7f4260152810 2024-03-28T15:01:53.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.717+0000 7f4264f7f640 1 --2- 172.21.15.186:0/387345566 >> v2:172.21.15.191:3300/0 conn(0x7f426006ce60 0x7f42600c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:53.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.718+0000 7f425ffff640 1 --2- 172.21.15.186:0/387345566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4260152810 0x7f4260158fc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f42600b2290 tx=0x7f425002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeab845a4251105f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:53.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.718+0000 7f425effd640 1 -- 172.21.15.186:0/387345566 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4250035020 con 0x7f4260152810 2024-03-28T15:01:53.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.718+0000 7f425effd640 1 -- 172.21.15.186:0/387345566 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4250031870 con 0x7f4260152810 2024-03-28T15:01:53.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.718+0000 7f425effd640 1 -- 172.21.15.186:0/387345566 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4250031c40 con 0x7f4260152810 2024-03-28T15:01:53.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.719+0000 7f42669e2640 1 -- 172.21.15.186:0/387345566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4260152810 msgr2=0x7f4260158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.719+0000 7f42669e2640 1 --2- 172.21.15.186:0/387345566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4260152810 0x7f4260158fc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f42600b2290 tx=0x7f425002b910 comp rx=0 tx=0).stop 2024-03-28T15:01:53.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.719+0000 7f42669e2640 1 -- 172.21.15.186:0/387345566 shutdown_connections 2024-03-28T15:01:53.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.719+0000 7f42669e2640 1 --2- 172.21.15.186:0/387345566 >> v2:172.21.15.191:3300/0 conn(0x7f426006ce60 0x7f42600c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.719+0000 7f42669e2640 1 --2- 172.21.15.186:0/387345566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4260071340 0x7f426006c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.719+0000 7f42669e2640 1 --2- 172.21.15.186:0/387345566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4260152810 0x7f4260158fc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.719+0000 7f42669e2640 1 -- 172.21.15.186:0/387345566 >> 172.21.15.186:0/387345566 conn(0x7f42600c8a20 msgr2=0x7f42600c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:53.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.719+0000 7f42669e2640 1 -- 172.21.15.186:0/387345566 shutdown_connections 2024-03-28T15:01:53.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.719+0000 7f42669e2640 1 -- 172.21.15.186:0/387345566 wait complete. 2024-03-28T15:01:53.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.720+0000 7f42669e2640 1 Processor -- start 2024-03-28T15:01:53.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.720+0000 7f42669e2640 1 -- start start 2024-03-28T15:01:53.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f42669e2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f426006ce60 0x7f42600cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f42669e2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4260071340 0x7f426006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f42669e2640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4260152810 0x7f426006da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f42669e2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f42600725f0 con 0x7f4260071340 2024-03-28T15:01:53.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f42669e2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4260072310 con 0x7f426006ce60 2024-03-28T15:01:53.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f42669e2640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f42600728d0 con 0x7f4260152810 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f425f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4260071340 0x7f426006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f4264f7f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4260152810 0x7f426006da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f425f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4260071340 0x7f426006d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42514/0 (socket says 172.21.15.186:42514) 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f425ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f426006ce60 0x7f42600cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f425f7fe640 1 -- 172.21.15.186:0/4147717666 learned_addr learned my addr 172.21.15.186:0/4147717666 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f425f7fe640 1 -- 172.21.15.186:0/4147717666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4260152810 msgr2=0x7f426006da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f425f7fe640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4260152810 0x7f426006da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f425f7fe640 1 -- 172.21.15.186:0/4147717666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f426006ce60 msgr2=0x7f42600cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f425f7fe640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f426006ce60 0x7f42600cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f425f7fe640 1 -- 172.21.15.186:0/4147717666 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42601576b0 con 0x7f4260071340 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f4264f7f640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4260152810 0x7f426006da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f425ffff640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f426006ce60 0x7f42600cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.721+0000 7f425f7fe640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4260071340 0x7f426006d500 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f425400e840 tx=0x7f425400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.722+0000 7f425cff9640 1 -- 172.21.15.186:0/4147717666 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4254017070 con 0x7f4260071340 2024-03-28T15:01:53.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.722+0000 7f425cff9640 1 -- 172.21.15.186:0/4147717666 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f425400c730 con 0x7f4260071340 2024-03-28T15:01:53.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.722+0000 7f42669e2640 1 -- 172.21.15.186:0/4147717666 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42601ee4d0 con 0x7f4260071340 2024-03-28T15:01:53.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.722+0000 7f425cff9640 1 -- 172.21.15.186:0/4147717666 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425400cb00 con 0x7f4260071340 2024-03-28T15:01:53.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.722+0000 7f42669e2640 1 -- 172.21.15.186:0/4147717666 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42601eea90 con 0x7f4260071340 2024-03-28T15:01:53.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.723+0000 7f42669e2640 1 -- 172.21.15.186:0/4147717666 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42601ee2c0 con 0x7f4260071340 2024-03-28T15:01:53.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.723+0000 7f425cff9640 1 -- 172.21.15.186:0/4147717666 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f425401a070 con 0x7f4260071340 2024-03-28T15:01:53.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.724+0000 7f425cff9640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f424c06c530 0x7f424c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:53.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.724+0000 7f425ffff640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f424c06c530 0x7f424c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:53.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.724+0000 7f425cff9640 1 -- 172.21.15.186:0/4147717666 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f425408b460 con 0x7f4260071340 2024-03-28T15:01:53.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.724+0000 7f425ffff640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f424c06c530 0x7f424c06e9f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4250036000 tx=0x7f425003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:53.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.728+0000 7f425cff9640 1 -- 172.21.15.186:0/4147717666 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f42540593a0 con 0x7f4260071340 2024-03-28T15:01:53.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.904+0000 7f42669e2640 1 -- 172.21.15.186:0/4147717666 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4260069500 con 0x7f4260071340 2024-03-28T15:01:53.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.906+0000 7f425cff9640 1 -- 172.21.15.186:0/4147717666 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f425405d6a0 con 0x7f4260071340 2024-03-28T15:01:53.906 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:53.906 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:01:33.099254+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:01:53.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 -- 172.21.15.186:0/4147717666 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f424c06c530 msgr2=0x7f424c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f424c06c530 0x7f424c06e9f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4250036000 tx=0x7f425003c040 comp rx=0 tx=0).stop 2024-03-28T15:01:53.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 -- 172.21.15.186:0/4147717666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4260071340 msgr2=0x7f426006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:53.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4260071340 0x7f426006d500 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f425400e840 tx=0x7f425400ec40 comp rx=0 tx=0).stop 2024-03-28T15:01:53.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 -- 172.21.15.186:0/4147717666 shutdown_connections 2024-03-28T15:01:53.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f424c06c530 0x7f424c06e9f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4260152810 0x7f426006da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4260071340 0x7f426006d500 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 --2- 172.21.15.186:0/4147717666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f426006ce60 0x7f42600cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:53.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 -- 172.21.15.186:0/4147717666 >> 172.21.15.186:0/4147717666 conn(0x7f42600c8a20 msgr2=0x7f426006ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:53.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 -- 172.21.15.186:0/4147717666 shutdown_connections 2024-03-28T15:01:53.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:53.908+0000 7f42669e2640 1 -- 172.21.15.186:0/4147717666 wait complete. 2024-03-28T15:01:53.919 DEBUG:tasks.mds_thrash:Yielding 2024-03-28T15:01:53.920 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T15:01:53.939 INFO:tasks.workunit:Pulling workunits from ref e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T15:01:53.940 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T15:01:53.940 INFO:tasks.workunit:timeout=6h 2024-03-28T15:01:53.940 INFO:tasks.workunit:cleanup=True 2024-03-28T15:01:53.940 DEBUG:teuthology.orchestra.run.smithi186:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T15:01:53.941 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2024-03-28T15:01:53.941 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before thrashing 2024-03-28T15:01:53.957 INFO:teuthology.orchestra.run.smithi186.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T15:01:53.957 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-28T15:01:53.957 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 2bh/43d Inode: 1 Links: 2 2024-03-28T15:01:53.957 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:01:53.957 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-28T15:01:53.957 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-28T15:01:53.957 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-28 15:01:37.114109299 +0000 2024-03-28T15:01:53.957 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-28 15:01:48.083091259 +0000 2024-03-28T15:01:53.957 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2024-03-28T15:01:53.958 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T15:01:53.958 DEBUG:teuthology.orchestra.run.smithi186:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T15:01:54.030 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T15:01:54.085 INFO:tasks.workunit.client.0.smithi186.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T15:01:55.232 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:01:55.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:01:55.942 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:01:56.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.054+0000 7f1083881640 1 Processor -- start 2024-03-28T15:01:56.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.054+0000 7f1083881640 1 -- start start 2024-03-28T15:01:56.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1083881640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f107c152960 0x7f107c0c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1083881640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f107c0723f0 0x7f107c06b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1083881640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f107c072810 0x7f107c06bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1083881640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f107c0c7550 con 0x7f107c072810 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1083881640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f107c158880 con 0x7f107c152960 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1083881640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f107c158b60 con 0x7f107c0723f0 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1081e1e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f107c072810 0x7f107c06bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1080e1c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f107c0723f0 0x7f107c06b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1081e1e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f107c072810 0x7f107c06bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42528/0 (socket says 172.21.15.186:42528) 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1081e1e640 1 -- 172.21.15.186:0/2907408958 learned_addr learned my addr 172.21.15.186:0/2907408958 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f108161d640 1 --2- 172.21.15.186:0/2907408958 >> v2:172.21.15.191:3300/0 conn(0x7f107c152960 0x7f107c0c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1080e1c640 1 -- 172.21.15.186:0/2907408958 >> v2:172.21.15.191:3300/0 conn(0x7f107c152960 msgr2=0x7f107c0c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1080e1c640 1 --2- 172.21.15.186:0/2907408958 >> v2:172.21.15.191:3300/0 conn(0x7f107c152960 0x7f107c0c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1080e1c640 1 -- 172.21.15.186:0/2907408958 >> v2:172.21.15.186:3300/0 conn(0x7f107c072810 msgr2=0x7f107c06bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:56.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1080e1c640 1 --2- 172.21.15.186:0/2907408958 >> v2:172.21.15.186:3300/0 conn(0x7f107c072810 0x7f107c06bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.057 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1080e1c640 1 -- 172.21.15.186:0/2907408958 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107c158e40 con 0x7f107c0723f0 2024-03-28T15:01:56.057 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.055+0000 7f1081e1e640 1 --2- 172.21.15.186:0/2907408958 >> v2:172.21.15.186:3300/0 conn(0x7f107c072810 0x7f107c06bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:56.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.056+0000 7f1080e1c640 1 --2- 172.21.15.186:0/2907408958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f107c0723f0 0x7f107c06b580 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f107c0caae0 tx=0x7f106c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=767e1985c784b486 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:56.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.056+0000 7f1073fff640 1 -- 172.21.15.186:0/2907408958 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106c0315a0 con 0x7f107c0723f0 2024-03-28T15:01:56.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.056+0000 7f1073fff640 1 -- 172.21.15.186:0/2907408958 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f106c037070 con 0x7f107c0723f0 2024-03-28T15:01:56.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.056+0000 7f1073fff640 1 -- 172.21.15.186:0/2907408958 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106c031960 con 0x7f107c0723f0 2024-03-28T15:01:56.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.057+0000 7f1083881640 1 -- 172.21.15.186:0/2907408958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f107c0723f0 msgr2=0x7f107c06b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:56.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.057+0000 7f1083881640 1 --2- 172.21.15.186:0/2907408958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f107c0723f0 0x7f107c06b580 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f107c0caae0 tx=0x7f106c02b840 comp rx=0 tx=0).stop 2024-03-28T15:01:56.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.057+0000 7f1083881640 1 -- 172.21.15.186:0/2907408958 shutdown_connections 2024-03-28T15:01:56.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.057+0000 7f1083881640 1 --2- 172.21.15.186:0/2907408958 >> v2:172.21.15.186:3300/0 conn(0x7f107c072810 0x7f107c06bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.057+0000 7f1083881640 1 --2- 172.21.15.186:0/2907408958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f107c0723f0 0x7f107c06b580 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.057+0000 7f1083881640 1 --2- 172.21.15.186:0/2907408958 >> v2:172.21.15.191:3300/0 conn(0x7f107c152960 0x7f107c0c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.057+0000 7f1083881640 1 -- 172.21.15.186:0/2907408958 >> 172.21.15.186:0/2907408958 conn(0x7f107c06e970 msgr2=0x7f107c06ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:56.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.057+0000 7f1083881640 1 -- 172.21.15.186:0/2907408958 shutdown_connections 2024-03-28T15:01:56.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.057+0000 7f1083881640 1 -- 172.21.15.186:0/2907408958 wait complete. 2024-03-28T15:01:56.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.058+0000 7f1083881640 1 Processor -- start 2024-03-28T15:01:56.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.058+0000 7f1083881640 1 -- start start 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1083881640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f107c0723f0 0x7f107c1f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1083881640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f107c072810 0x7f107c06b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1083881640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f107c152960 0x7f107c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1083881640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f107c152d40 con 0x7f107c072810 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1083881640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f107c159400 con 0x7f107c0723f0 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1083881640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f107c152ec0 con 0x7f107c152960 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1080e1c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f107c072810 0x7f107c06b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1080e1c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f107c072810 0x7f107c06b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42540/0 (socket says 172.21.15.186:42540) 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1080e1c640 1 -- 172.21.15.186:0/4189966491 learned_addr learned my addr 172.21.15.186:0/4189966491 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f108161d640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f107c0723f0 0x7f107c1f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1081e1e640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f107c152960 0x7f107c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1080e1c640 1 -- 172.21.15.186:0/4189966491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f107c152960 msgr2=0x7f107c06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:56.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1080e1c640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f107c152960 0x7f107c06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1080e1c640 1 -- 172.21.15.186:0/4189966491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f107c0723f0 msgr2=0x7f107c1f4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:56.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1080e1c640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f107c0723f0 0x7f107c1f4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1080e1c640 1 -- 172.21.15.186:0/4189966491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107c0c4920 con 0x7f107c072810 2024-03-28T15:01:56.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f108161d640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f107c0723f0 0x7f107c1f4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:56.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1080e1c640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f107c072810 0x7f107c06b460 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f106c000c00 tx=0x7f106c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:56.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1071ffb640 1 -- 172.21.15.186:0/4189966491 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106c032630 con 0x7f107c072810 2024-03-28T15:01:56.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1071ffb640 1 -- 172.21.15.186:0/4189966491 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f106c0327f0 con 0x7f107c072810 2024-03-28T15:01:56.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.059+0000 7f1083881640 1 -- 172.21.15.186:0/4189966491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f107c0c3270 con 0x7f107c072810 2024-03-28T15:01:56.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.060+0000 7f1071ffb640 1 -- 172.21.15.186:0/4189966491 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106c032be0 con 0x7f107c072810 2024-03-28T15:01:56.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.060+0000 7f1083881640 1 -- 172.21.15.186:0/4189966491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f107c1580b0 con 0x7f107c072810 2024-03-28T15:01:56.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.061+0000 7f1083881640 1 -- 172.21.15.186:0/4189966491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f107c069500 con 0x7f107c072810 2024-03-28T15:01:56.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.061+0000 7f1071ffb640 1 -- 172.21.15.186:0/4189966491 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f106c04d3f0 con 0x7f107c072810 2024-03-28T15:01:56.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.062+0000 7f1071ffb640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f106006c560 0x7f106006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.062+0000 7f1071ffb640 1 -- 172.21.15.186:0/4189966491 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f106c02f030 con 0x7f107c072810 2024-03-28T15:01:56.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.062+0000 7f108161d640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f106006c560 0x7f106006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.062+0000 7f108161d640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f106006c560 0x7f106006ea20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1078009f30 tx=0x7f1078009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:56.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.066+0000 7f1071ffb640 1 -- 172.21.15.186:0/4189966491 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f107c0c3270 con 0x7f107c072810 2024-03-28T15:01:56.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.281+0000 7f1083881640 1 -- 172.21.15.186:0/4189966491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f107c0701c0 con 0x7f107c072810 2024-03-28T15:01:56.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.282+0000 7f1071ffb640 1 -- 172.21.15.186:0/4189966491 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7f107c0c4920 con 0x7f107c072810 2024-03-28T15:01:56.282 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:56.282 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:01:56.283 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:01:56.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 -- 172.21.15.186:0/4189966491 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f106006c560 msgr2=0x7f106006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:56.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f106006c560 0x7f106006ea20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1078009f30 tx=0x7f1078009190 comp rx=0 tx=0).stop 2024-03-28T15:01:56.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 -- 172.21.15.186:0/4189966491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f107c072810 msgr2=0x7f107c06b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:56.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f107c072810 0x7f107c06b460 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f106c000c00 tx=0x7f106c0046c0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 -- 172.21.15.186:0/4189966491 shutdown_connections 2024-03-28T15:01:56.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f106006c560 0x7f106006ea20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f107c152960 0x7f107c06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f107c072810 0x7f107c06b460 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 --2- 172.21.15.186:0/4189966491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f107c0723f0 0x7f107c1f4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 -- 172.21.15.186:0/4189966491 >> 172.21.15.186:0/4189966491 conn(0x7f107c06e970 msgr2=0x7f107c14ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:56.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 -- 172.21.15.186:0/4189966491 shutdown_connections 2024-03-28T15:01:56.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.284+0000 7f1083881640 1 -- 172.21.15.186:0/4189966491 wait complete. 2024-03-28T15:01:56.295 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2024-03-28T15:01:56.295 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-03-28T15:01:56.295 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:01:56.305 INFO:tasks.ceph.mds.f:Stopped 2024-03-28T15:01:56.305 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:01:56.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.405+0000 7f52d9ba0640 1 Processor -- start 2024-03-28T15:01:56.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.405+0000 7f52d9ba0640 1 -- start start 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d9ba0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52d4071340 0x7f52d4158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d9ba0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d40c8090 0x7f52d40c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d9ba0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52d4072df0 con 0x7f52d4152810 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d9ba0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f52d4159500 con 0x7f52d4071340 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d9ba0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f52d40c8660 con 0x7f52d40c8090 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52cbfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52d4071340 0x7f52d4158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d3fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d40c8090 0x7f52d40c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52cbfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52d4071340 0x7f52d4158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:36844/0 (socket says 172.21.15.186:36844) 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d37fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f52d4152810 0x7f52d406c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:59386/0 (socket says 172.21.15.186:59386) 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d3fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d40c8090 0x7f52d40c6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:46960/0 (socket says 172.21.15.186:46960) 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52cbfff640 1 -- 172.21.15.186:0/1854971213 learned_addr learned my addr 172.21.15.186:0/1854971213 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d3fff640 1 -- 172.21.15.186:0/1854971213 >> v2:172.21.15.191:3300/0 conn(0x7f52d4071340 msgr2=0x7f52d4158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:56.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d3fff640 1 --2- 172.21.15.186:0/1854971213 >> v2:172.21.15.191:3300/0 conn(0x7f52d4071340 0x7f52d4158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d3fff640 1 -- 172.21.15.186:0/1854971213 >> v1:172.21.15.191:6789/0 conn(0x7f52d4152810 legacy=0x7f52d406c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:56.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d2ffd640 1 -- 172.21.15.186:0/1854971213 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1189751657 0 0) 0x7f52d4072df0 con 0x7f52d4152810 2024-03-28T15:01:56.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52d3fff640 1 -- 172.21.15.186:0/1854971213 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d4071d50 con 0x7f52d40c8090 2024-03-28T15:01:56.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.406+0000 7f52cbfff640 1 --2- 172.21.15.186:0/1854971213 >> v2:172.21.15.191:3300/0 conn(0x7f52d4071340 0x7f52d4158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:56.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.407+0000 7f52d3fff640 1 --2- 172.21.15.186:0/1854971213 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d40c8090 0x7f52d40c6cb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009d90 tx=0x7f52cc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f02201a6d0db8c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:56.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.407+0000 7f52d2ffd640 1 -- 172.21.15.186:0/1854971213 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52cc039070 con 0x7f52d40c8090 2024-03-28T15:01:56.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.407+0000 7f52d2ffd640 1 -- 172.21.15.186:0/1854971213 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52cc02cc50 con 0x7f52d40c8090 2024-03-28T15:01:56.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.407+0000 7f52d2ffd640 1 -- 172.21.15.186:0/1854971213 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52cc034440 con 0x7f52d40c8090 2024-03-28T15:01:56.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.408+0000 7f52d9ba0640 1 -- 172.21.15.186:0/1854971213 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d40c8090 msgr2=0x7f52d40c6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:56.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.408+0000 7f52d9ba0640 1 --2- 172.21.15.186:0/1854971213 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d40c8090 0x7f52d40c6cb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009d90 tx=0x7f52cc02bea0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.408+0000 7f52d9ba0640 1 -- 172.21.15.186:0/1854971213 shutdown_connections 2024-03-28T15:01:56.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.408+0000 7f52d9ba0640 1 --2- 172.21.15.186:0/1854971213 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d40c8090 0x7f52d40c6cb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.408+0000 7f52d9ba0640 1 --2- 172.21.15.186:0/1854971213 >> v2:172.21.15.191:3300/0 conn(0x7f52d4071340 0x7f52d4158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.408+0000 7f52d9ba0640 1 -- 172.21.15.186:0/1854971213 >> 172.21.15.186:0/1854971213 conn(0x7f52d40c8a20 msgr2=0x7f52d40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:56.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.408+0000 7f52d9ba0640 1 -- 172.21.15.186:0/1854971213 shutdown_connections 2024-03-28T15:01:56.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.408+0000 7f52d9ba0640 1 -- 172.21.15.186:0/1854971213 wait complete. 2024-03-28T15:01:56.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.409+0000 7f52d9ba0640 1 Processor -- start 2024-03-28T15:01:56.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.409+0000 7f52d9ba0640 1 -- start start 2024-03-28T15:01:56.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.409+0000 7f52d9ba0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d4071340 0x7f52d4150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.409+0000 7f52d9ba0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d40c8090 0x7f52d414fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.409+0000 7f52d9ba0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d4152810 0x7f52d4150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.409+0000 7f52d9ba0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52d40725f0 con 0x7f52d40c8090 2024-03-28T15:01:56.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.409+0000 7f52d9ba0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f52d4072310 con 0x7f52d4071340 2024-03-28T15:01:56.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.409+0000 7f52d9ba0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f52d40728d0 con 0x7f52d4152810 2024-03-28T15:01:56.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52cbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d40c8090 0x7f52d414fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52cbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d40c8090 0x7f52d414fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42544/0 (socket says 172.21.15.186:42544) 2024-03-28T15:01:56.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52cbfff640 1 -- 172.21.15.186:0/2213645073 learned_addr learned my addr 172.21.15.186:0/2213645073 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:56.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52d37fe640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d4071340 0x7f52d4150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52d3fff640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d4152810 0x7f52d4150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52cbfff640 1 -- 172.21.15.186:0/2213645073 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d4152810 msgr2=0x7f52d4150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52cbfff640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d4152810 0x7f52d4150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52cbfff640 1 -- 172.21.15.186:0/2213645073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d4071340 msgr2=0x7f52d4150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52cbfff640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d4071340 0x7f52d4150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52cbfff640 1 -- 172.21.15.186:0/2213645073 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d40c5900 con 0x7f52d40c8090 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52d37fe640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d4071340 0x7f52d4150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52cbfff640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d40c8090 0x7f52d414fe20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00e900 tx=0x7f52bc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52d0ff9640 1 -- 172.21.15.186:0/2213645073 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc017070 con 0x7f52d40c8090 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52d0ff9640 1 -- 172.21.15.186:0/2213645073 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52bc00ca00 con 0x7f52d40c8090 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52d0ff9640 1 -- 172.21.15.186:0/2213645073 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc00cdf0 con 0x7f52d40c8090 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52d9ba0640 1 -- 172.21.15.186:0/2213645073 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d41ee4d0 con 0x7f52d40c8090 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.410+0000 7f52d9ba0640 1 -- 172.21.15.186:0/2213645073 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d41eea90 con 0x7f52d40c8090 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.411+0000 7f52d9ba0640 1 -- 172.21.15.186:0/2213645073 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d40c4fb0 con 0x7f52d40c8090 2024-03-28T15:01:56.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.412+0000 7f52d0ff9640 1 -- 172.21.15.186:0/2213645073 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f52bc01a070 con 0x7f52d40c8090 2024-03-28T15:01:56.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.412+0000 7f52d0ff9640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52b406c560 0x7f52b406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:56.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.412+0000 7f52d0ff9640 1 -- 172.21.15.186:0/2213645073 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f52bc08b8d0 con 0x7f52d40c8090 2024-03-28T15:01:56.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.413+0000 7f52d37fe640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52b406c560 0x7f52b406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:56.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.413+0000 7f52d37fe640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52b406c560 0x7f52b406ea20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f52c4000f60 tx=0x7f52c4007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:56.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.417+0000 7f52d0ff9640 1 -- 172.21.15.186:0/2213645073 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f52d40c4fb0 con 0x7f52d40c8090 2024-03-28T15:01:56.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.629+0000 7f52d9ba0640 1 -- 172.21.15.186:0/2213645073 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52d4069500 con 0x7f52d40c8090 2024-03-28T15:01:56.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.630+0000 7f52d0ff9640 1 -- 172.21.15.186:0/2213645073 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7f52bc059810 con 0x7f52d40c8090 2024-03-28T15:01:56.630 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:56.631 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:01:56.631 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:01:56.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 -- 172.21.15.186:0/2213645073 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52b406c560 msgr2=0x7f52b406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:56.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52b406c560 0x7f52b406ea20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f52c4000f60 tx=0x7f52c4007d50 comp rx=0 tx=0).stop 2024-03-28T15:01:56.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 -- 172.21.15.186:0/2213645073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d40c8090 msgr2=0x7f52d414fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:56.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d40c8090 0x7f52d414fe20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00e900 tx=0x7f52bc00edc0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 -- 172.21.15.186:0/2213645073 shutdown_connections 2024-03-28T15:01:56.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52b406c560 0x7f52b406ea20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52d4152810 0x7f52d4150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52d40c8090 0x7f52d414fe20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 --2- 172.21.15.186:0/2213645073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d4071340 0x7f52d4150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:56.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 -- 172.21.15.186:0/2213645073 >> 172.21.15.186:0/2213645073 conn(0x7f52d40c8a20 msgr2=0x7f52d40ca800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:56.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 -- 172.21.15.186:0/2213645073 shutdown_connections 2024-03-28T15:01:56.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:56.632+0000 7f52d9ba0640 1 -- 172.21.15.186:0/2213645073 wait complete. 2024-03-28T15:01:56.643 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:01:58.645 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:01:58.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.757+0000 7f86abf1e640 1 Processor -- start 2024-03-28T15:01:58.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.757+0000 7f86abf1e640 1 -- start start 2024-03-28T15:01:58.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86abf1e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f86a4152810 0x7f86a4158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:58.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86abf1e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a4071340 0x7f86a406c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:58.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86abf1e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86a406ce60 0x7f86a40c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:58.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86abf1e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f86a4159500 con 0x7f86a406ce60 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86abf1e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f86a4072df0 con 0x7f86a4152810 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86abf1e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f86a40c8660 con 0x7f86a4071340 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86aa4bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86a406ce60 0x7f86a40c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86aa4bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86a406ce60 0x7f86a40c6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36350/0 (socket says 172.21.15.186:36350) 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86aa4bb640 1 -- 172.21.15.186:0/2682862901 learned_addr learned my addr 172.21.15.186:0/2682862901 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86a9cba640 1 --2- 172.21.15.186:0/2682862901 >> v2:172.21.15.191:3300/0 conn(0x7f86a4152810 0x7f86a4158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86a94b9640 1 --2- 172.21.15.186:0/2682862901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a4071340 0x7f86a406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86aa4bb640 1 -- 172.21.15.186:0/2682862901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a4071340 msgr2=0x7f86a406c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86aa4bb640 1 --2- 172.21.15.186:0/2682862901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a4071340 0x7f86a406c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86aa4bb640 1 -- 172.21.15.186:0/2682862901 >> v2:172.21.15.191:3300/0 conn(0x7f86a4152810 msgr2=0x7f86a4158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86aa4bb640 1 --2- 172.21.15.186:0/2682862901 >> v2:172.21.15.191:3300/0 conn(0x7f86a4152810 0x7f86a4158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86aa4bb640 1 -- 172.21.15.186:0/2682862901 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a4071d50 con 0x7f86a406ce60 2024-03-28T15:01:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.758+0000 7f86a94b9640 1 --2- 172.21.15.186:0/2682862901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a4071340 0x7f86a406c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.759+0000 7f86aa4bb640 1 --2- 172.21.15.186:0/2682862901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a406ce60 0x7f86a40c6cb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f86a40c31c0 tx=0x7f869c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887ffe30f414e4b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.759+0000 7f86a8cb8640 1 -- 172.21.15.186:0/2682862901 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c02ca70 con 0x7f86a406ce60 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.759+0000 7f86a8cb8640 1 -- 172.21.15.186:0/2682862901 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f869c02cc10 con 0x7f86a406ce60 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.759+0000 7f86a8cb8640 1 -- 172.21.15.186:0/2682862901 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c0343f0 con 0x7f86a406ce60 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.759+0000 7f86abf1e640 1 -- 172.21.15.186:0/2682862901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a406ce60 msgr2=0x7f86a40c6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.759+0000 7f86abf1e640 1 --2- 172.21.15.186:0/2682862901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a406ce60 0x7f86a40c6cb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f86a40c31c0 tx=0x7f869c02bed0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.760+0000 7f86abf1e640 1 -- 172.21.15.186:0/2682862901 shutdown_connections 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.760+0000 7f86abf1e640 1 --2- 172.21.15.186:0/2682862901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a406ce60 0x7f86a40c6cb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.760+0000 7f86abf1e640 1 --2- 172.21.15.186:0/2682862901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a4071340 0x7f86a406c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.760+0000 7f86abf1e640 1 --2- 172.21.15.186:0/2682862901 >> v2:172.21.15.191:3300/0 conn(0x7f86a4152810 0x7f86a4158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.760+0000 7f86abf1e640 1 -- 172.21.15.186:0/2682862901 >> 172.21.15.186:0/2682862901 conn(0x7f86a40c8a20 msgr2=0x7f86a40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.760+0000 7f86abf1e640 1 -- 172.21.15.186:0/2682862901 shutdown_connections 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.760+0000 7f86abf1e640 1 -- 172.21.15.186:0/2682862901 wait complete. 2024-03-28T15:01:58.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.761+0000 7f86abf1e640 1 Processor -- start 2024-03-28T15:01:58.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.761+0000 7f86abf1e640 1 -- start start 2024-03-28T15:01:58.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.761+0000 7f86abf1e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a406ce60 0x7f86a414d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:58.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.761+0000 7f86abf1e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4071340 0x7f86a40ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:58.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.761+0000 7f86abf1e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a4152810 0x7f86a40ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:58.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.761+0000 7f86abf1e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86a40725f0 con 0x7f86a4152810 2024-03-28T15:01:58.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.761+0000 7f86abf1e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f86a4072310 con 0x7f86a4071340 2024-03-28T15:01:58.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.761+0000 7f86abf1e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f86a40728d0 con 0x7f86a406ce60 2024-03-28T15:01:58.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86aa4bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a4152810 0x7f86a40ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:58.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86aa4bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a4152810 0x7f86a40ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36364/0 (socket says 172.21.15.186:36364) 2024-03-28T15:01:58.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86aa4bb640 1 -- 172.21.15.186:0/3635228638 learned_addr learned my addr 172.21.15.186:0/3635228638 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:01:58.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86a9cba640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a406ce60 0x7f86a414d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:58.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86a94b9640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4071340 0x7f86a40ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86aa4bb640 1 -- 172.21.15.186:0/3635228638 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a406ce60 msgr2=0x7f86a414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86aa4bb640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a406ce60 0x7f86a414d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86aa4bb640 1 -- 172.21.15.186:0/3635228638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4071340 msgr2=0x7f86a40ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86aa4bb640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4071340 0x7f86a40ca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86aa4bb640 1 -- 172.21.15.186:0/3635228638 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a414ebe0 con 0x7f86a4152810 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86aa4bb640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a4152810 0x7f86a40ca600 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f869c009f40 tx=0x7f869c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f869a7fc640 1 -- 172.21.15.186:0/3635228638 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c043020 con 0x7f86a4152810 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f869a7fc640 1 -- 172.21.15.186:0/3635228638 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f869c03e450 con 0x7f86a4152810 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86abf1e640 1 -- 172.21.15.186:0/3635228638 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a41f4300 con 0x7f86a4152810 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f869a7fc640 1 -- 172.21.15.186:0/3635228638 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c009510 con 0x7f86a4152810 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.762+0000 7f86abf1e640 1 -- 172.21.15.186:0/3635228638 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a414bf70 con 0x7f86a4152810 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.763+0000 7f86abf1e640 1 -- 172.21.15.186:0/3635228638 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a4069500 con 0x7f86a4152810 2024-03-28T15:01:58.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.764+0000 7f869a7fc640 1 -- 172.21.15.186:0/3635228638 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f869c046020 con 0x7f86a4152810 2024-03-28T15:01:58.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.764+0000 7f869a7fc640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f868806c530 0x7f868806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:01:58.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.764+0000 7f869a7fc640 1 -- 172.21.15.186:0/3635228638 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f869c0ae150 con 0x7f86a4152810 2024-03-28T15:01:58.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.764+0000 7f86a9cba640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f868806c530 0x7f868806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:01:58.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.765+0000 7f86a9cba640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f868806c530 0x7f868806e9f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f86a0007920 tx=0x7f86a0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:01:58.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.769+0000 7f869a7fc640 1 -- 172.21.15.186:0/3635228638 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f869c04d020 con 0x7f86a4152810 2024-03-28T15:01:58.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.983+0000 7f86abf1e640 1 -- 172.21.15.186:0/3635228638 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86a40c77f0 con 0x7f86a4152810 2024-03-28T15:01:58.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.983+0000 7f869a7fc640 1 -- 172.21.15.186:0/3635228638 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7f86a414ebe0 con 0x7f86a4152810 2024-03-28T15:01:58.984 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:01:58.984 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:01:58.984 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.985+0000 7f86abf1e640 1 -- 172.21.15.186:0/3635228638 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f868806c530 msgr2=0x7f868806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.985+0000 7f86abf1e640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f868806c530 0x7f868806e9f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f86a0007920 tx=0x7f86a0008040 comp rx=0 tx=0).stop 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.985+0000 7f86abf1e640 1 -- 172.21.15.186:0/3635228638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a4152810 msgr2=0x7f86a40ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.985+0000 7f86abf1e640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a4152810 0x7f86a40ca600 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f869c009f40 tx=0x7f869c004a40 comp rx=0 tx=0).stop 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.985+0000 7f86abf1e640 1 -- 172.21.15.186:0/3635228638 shutdown_connections 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.985+0000 7f86abf1e640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f868806c530 0x7f868806e9f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.985+0000 7f86abf1e640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a4152810 0x7f86a40ca600 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.985+0000 7f86abf1e640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4071340 0x7f86a40ca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.985+0000 7f86abf1e640 1 --2- 172.21.15.186:0/3635228638 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a406ce60 0x7f86a414d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.985+0000 7f86abf1e640 1 -- 172.21.15.186:0/3635228638 >> 172.21.15.186:0/3635228638 conn(0x7f86a40c8a20 msgr2=0x7f86a406f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.986+0000 7f86abf1e640 1 -- 172.21.15.186:0/3635228638 shutdown_connections 2024-03-28T15:01:58.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:01:58.986+0000 7f86abf1e640 1 -- 172.21.15.186:0/3635228638 wait complete. 2024-03-28T15:01:58.997 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:02:00.998 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.111+0000 7f497273e640 1 Processor -- start 2024-03-28T15:02:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.111+0000 7f497273e640 1 -- start start 2024-03-28T15:02:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.112+0000 7f497273e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f496c151990 0x7f496c0c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.112+0000 7f497273e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f496c06caa0 con 0x7f496c0b1e30 2024-03-28T15:02:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.112+0000 7f497273e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f496c06cd80 con 0x7f496c0c2d00 2024-03-28T15:02:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.112+0000 7f497273e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f496c0c1da0 con 0x7f496c151990 2024-03-28T15:02:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.112+0000 7f496bfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f496c151990 0x7f496c0c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.112+0000 7f496b7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f496c0b1e30 0x7f496c0712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:47068/0 (socket says 172.21.15.186:47068) 2024-03-28T15:02:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.112+0000 7f496b7fe640 1 -- 172.21.15.186:0/2014737107 learned_addr learned my addr 172.21.15.186:0/2014737107 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:02:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.112+0000 7f496bfff640 1 -- 172.21.15.186:0/2014737107 >> v1:172.21.15.191:6789/0 conn(0x7f496c0c2d00 legacy=0x7f496c161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.112+0000 7f496bfff640 1 -- 172.21.15.186:0/2014737107 >> v1:172.21.15.186:6789/0 conn(0x7f496c0b1e30 legacy=0x7f496c0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.112+0000 7f496bfff640 1 -- 172.21.15.186:0/2014737107 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c0c1340 con 0x7f496c151990 2024-03-28T15:02:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.112+0000 7f496affd640 1 -- 172.21.15.186:0/2014737107 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 17220243 0 0) 0x7f496c06caa0 con 0x7f496c0b1e30 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.113+0000 7f496bfff640 1 --2- 172.21.15.186:0/2014737107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f496c151990 0x7f496c0c2230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f495c0097a0 tx=0x7f495c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4eb6fe6a6677fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.113+0000 7f496affd640 1 -- 172.21.15.186:0/2014737107 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c02f810 con 0x7f496c151990 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.113+0000 7f496affd640 1 -- 172.21.15.186:0/2014737107 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f495c02f9b0 con 0x7f496c151990 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.113+0000 7f496affd640 1 -- 172.21.15.186:0/2014737107 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c02fd80 con 0x7f496c151990 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.114+0000 7f497273e640 1 -- 172.21.15.186:0/2014737107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f496c151990 msgr2=0x7f496c0c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.114+0000 7f497273e640 1 --2- 172.21.15.186:0/2014737107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f496c151990 0x7f496c0c2230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f495c0097a0 tx=0x7f495c02b9f0 comp rx=0 tx=0).stop 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.114+0000 7f497273e640 1 -- 172.21.15.186:0/2014737107 shutdown_connections 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.114+0000 7f497273e640 1 --2- 172.21.15.186:0/2014737107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f496c151990 0x7f496c0c2230 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.114+0000 7f497273e640 1 -- 172.21.15.186:0/2014737107 >> 172.21.15.186:0/2014737107 conn(0x7f496c0c8a20 msgr2=0x7f496c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.114+0000 7f497273e640 1 -- 172.21.15.186:0/2014737107 shutdown_connections 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.114+0000 7f497273e640 1 -- 172.21.15.186:0/2014737107 wait complete. 2024-03-28T15:02:01.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.115+0000 7f497273e640 1 Processor -- start 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.115+0000 7f497273e640 1 -- start start 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f497273e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f496c0b1e30 0x7f496c14bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f497273e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f496c0c2d00 0x7f496c071420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f497273e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f496c151990 0x7f496c0c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f497273e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f496c073dd0 con 0x7f496c151990 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f497273e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f496c0c1900 con 0x7f496c0b1e30 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f497273e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f496c073f50 con 0x7f496c0c2d00 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f4970cdb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f496c151990 0x7f496c0c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f4970cdb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f496c151990 0x7f496c0c4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36388/0 (socket says 172.21.15.186:36388) 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f4970cdb640 1 -- 172.21.15.186:0/602376925 learned_addr learned my addr 172.21.15.186:0/602376925 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f496bfff640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f496c0b1e30 0x7f496c14bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f496b7fe640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f496c0c2d00 0x7f496c071420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f4970cdb640 1 -- 172.21.15.186:0/602376925 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f496c0c2d00 msgr2=0x7f496c071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:01.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f4970cdb640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f496c0c2d00 0x7f496c071420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:01.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f4970cdb640 1 -- 172.21.15.186:0/602376925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f496c0b1e30 msgr2=0x7f496c14bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:01.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f4970cdb640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f496c0b1e30 0x7f496c14bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:01.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f4970cdb640 1 -- 172.21.15.186:0/602376925 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c14dc50 con 0x7f496c151990 2024-03-28T15:02:01.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f496bfff640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f496c0b1e30 0x7f496c14bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:02:01.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.116+0000 7f4970cdb640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f496c151990 0x7f496c0c4a30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f496400c6b0 tx=0x7f496400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:01.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.117+0000 7f4968ff9640 1 -- 172.21.15.186:0/602376925 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964003b70 con 0x7f496c151990 2024-03-28T15:02:01.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.117+0000 7f4968ff9640 1 -- 172.21.15.186:0/602376925 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4964003d30 con 0x7f496c151990 2024-03-28T15:02:01.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.117+0000 7f4968ff9640 1 -- 172.21.15.186:0/602376925 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964005310 con 0x7f496c151990 2024-03-28T15:02:01.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.117+0000 7f497273e640 1 -- 172.21.15.186:0/602376925 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496c14ceb0 con 0x7f496c151990 2024-03-28T15:02:01.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.117+0000 7f497273e640 1 -- 172.21.15.186:0/602376925 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f496c0cad40 con 0x7f496c151990 2024-03-28T15:02:01.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.118+0000 7f497273e640 1 -- 172.21.15.186:0/602376925 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f496c069500 con 0x7f496c151990 2024-03-28T15:02:01.121 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.118+0000 7f4968ff9640 1 -- 172.21.15.186:0/602376925 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f49640054b0 con 0x7f496c151990 2024-03-28T15:02:01.121 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.119+0000 7f4968ff9640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f494c06c560 0x7f494c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:01.121 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.119+0000 7f4968ff9640 1 -- 172.21.15.186:0/602376925 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f496408b5a0 con 0x7f496c151990 2024-03-28T15:02:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.119+0000 7f496bfff640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f494c06c560 0x7f494c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:01.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.119+0000 7f496bfff640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f494c06c560 0x7f494c06ea20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f495c036000 tx=0x7f495c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:01.123 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.123+0000 7f4968ff9640 1 -- 172.21.15.186:0/602376925 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4964059b30 con 0x7f496c151990 2024-03-28T15:02:01.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.338+0000 7f497273e640 1 -- 172.21.15.186:0/602376925 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f496c1511b0 con 0x7f496c151990 2024-03-28T15:02:01.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.338+0000 7f4968ff9640 1 -- 172.21.15.186:0/602376925 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7f4964059430 con 0x7f496c151990 2024-03-28T15:02:01.339 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:01.339 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:02:01.339 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:02:01.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 -- 172.21.15.186:0/602376925 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f494c06c560 msgr2=0x7f494c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:01.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f494c06c560 0x7f494c06ea20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f495c036000 tx=0x7f495c008030 comp rx=0 tx=0).stop 2024-03-28T15:02:01.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 -- 172.21.15.186:0/602376925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f496c151990 msgr2=0x7f496c0c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:01.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f496c151990 0x7f496c0c4a30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f496400c6b0 tx=0x7f496400cb70 comp rx=0 tx=0).stop 2024-03-28T15:02:01.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 -- 172.21.15.186:0/602376925 shutdown_connections 2024-03-28T15:02:01.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f494c06c560 0x7f494c06ea20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:01.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f496c151990 0x7f496c0c4a30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:01.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f496c0c2d00 0x7f496c071420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:01.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 --2- 172.21.15.186:0/602376925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f496c0b1e30 0x7f496c14bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:01.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 -- 172.21.15.186:0/602376925 >> 172.21.15.186:0/602376925 conn(0x7f496c0c8a20 msgr2=0x7f496c0c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:01.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 -- 172.21.15.186:0/602376925 shutdown_connections 2024-03-28T15:02:01.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:01.340+0000 7f497273e640 1 -- 172.21.15.186:0/602376925 wait complete. 2024-03-28T15:02:01.351 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:02:03.353 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:03.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.465+0000 7f8667047640 1 Processor -- start 2024-03-28T15:02:03.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.465+0000 7f8667047640 1 -- start start 2024-03-28T15:02:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.466+0000 7f8667047640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8660151990 0x7f86600c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.466+0000 7f8667047640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86600b1e30 0x7f86600c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.466+0000 7f8667047640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86600c2d00 0x7f86600712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.466+0000 7f8667047640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f86600c1da0 con 0x7f86600b1e30 2024-03-28T15:02:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.466+0000 7f8667047640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f866006caa0 con 0x7f8660151990 2024-03-28T15:02:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.466+0000 7f8667047640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f866006cd80 con 0x7f86600c2d00 2024-03-28T15:02:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.466+0000 7f865ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86600b1e30 0x7f86600c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.466+0000 7f865ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86600b1e30 0x7f86600c2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36392/0 (socket says 172.21.15.186:36392) 2024-03-28T15:02:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.466+0000 7f865ffff640 1 -- 172.21.15.186:0/3291755013 learned_addr learned my addr 172.21.15.186:0/3291755013 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.467+0000 7f86655e4640 1 --2- 172.21.15.186:0/3291755013 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86600c2d00 0x7f86600712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.467+0000 7f8664de3640 1 --2- 172.21.15.186:0/3291755013 >> v2:172.21.15.191:3300/0 conn(0x7f8660151990 0x7f86600c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:03.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.467+0000 7f865ffff640 1 -- 172.21.15.186:0/3291755013 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86600c2d00 msgr2=0x7f86600712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:03.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.467+0000 7f865ffff640 1 --2- 172.21.15.186:0/3291755013 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86600c2d00 0x7f86600712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.467+0000 7f865ffff640 1 -- 172.21.15.186:0/3291755013 >> v2:172.21.15.191:3300/0 conn(0x7f8660151990 msgr2=0x7f86600c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:03.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.467+0000 7f865ffff640 1 --2- 172.21.15.186:0/3291755013 >> v2:172.21.15.191:3300/0 conn(0x7f8660151990 0x7f86600c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.467+0000 7f865ffff640 1 -- 172.21.15.186:0/3291755013 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86600c1340 con 0x7f86600b1e30 2024-03-28T15:02:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.467+0000 7f865ffff640 1 --2- 172.21.15.186:0/3291755013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86600b1e30 0x7f86600c2770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f866006fd90 tx=0x7f864c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1174a459bbd1e393 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f865f7fe640 1 -- 172.21.15.186:0/3291755013 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c034070 con 0x7f86600b1e30 2024-03-28T15:02:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f865f7fe640 1 -- 172.21.15.186:0/3291755013 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f864c030860 con 0x7f86600b1e30 2024-03-28T15:02:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f865f7fe640 1 -- 172.21.15.186:0/3291755013 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c030c30 con 0x7f86600b1e30 2024-03-28T15:02:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f8667047640 1 -- 172.21.15.186:0/3291755013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86600b1e30 msgr2=0x7f86600c2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f8667047640 1 --2- 172.21.15.186:0/3291755013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86600b1e30 0x7f86600c2770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f866006fd90 tx=0x7f864c02b9a0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f8667047640 1 -- 172.21.15.186:0/3291755013 shutdown_connections 2024-03-28T15:02:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f8667047640 1 --2- 172.21.15.186:0/3291755013 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86600c2d00 0x7f86600712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f8667047640 1 --2- 172.21.15.186:0/3291755013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86600b1e30 0x7f86600c2770 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f8667047640 1 --2- 172.21.15.186:0/3291755013 >> v2:172.21.15.191:3300/0 conn(0x7f8660151990 0x7f86600c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f8667047640 1 -- 172.21.15.186:0/3291755013 >> 172.21.15.186:0/3291755013 conn(0x7f86600c8a20 msgr2=0x7f86600c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:03.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f8667047640 1 -- 172.21.15.186:0/3291755013 shutdown_connections 2024-03-28T15:02:03.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.468+0000 7f8667047640 1 -- 172.21.15.186:0/3291755013 wait complete. 2024-03-28T15:02:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.469+0000 7f8667047640 1 Processor -- start 2024-03-28T15:02:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.469+0000 7f8667047640 1 -- start start 2024-03-28T15:02:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.470+0000 7f8667047640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86600b1e30 0x7f8660144980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.470+0000 7f8667047640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86600c2d00 0x7f86601f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.470+0000 7f8667047640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8660151990 0x7f86601eed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.470+0000 7f8667047640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8660073dd0 con 0x7f8660151990 2024-03-28T15:02:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.470+0000 7f8667047640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f86600c1900 con 0x7f86600c2d00 2024-03-28T15:02:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.470+0000 7f8667047640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8660073f50 con 0x7f86600b1e30 2024-03-28T15:02:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.470+0000 7f86655e4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8660151990 0x7f86601eed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.470+0000 7f86655e4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8660151990 0x7f86601eed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36400/0 (socket says 172.21.15.186:36400) 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.470+0000 7f86655e4640 1 -- 172.21.15.186:0/349093875 learned_addr learned my addr 172.21.15.186:0/349093875 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.470+0000 7f8664de3640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86600b1e30 0x7f8660144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.470+0000 7f865ffff640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86600c2d00 0x7f86601f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.471+0000 7f86655e4640 1 -- 172.21.15.186:0/349093875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86600b1e30 msgr2=0x7f8660144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.471+0000 7f86655e4640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86600b1e30 0x7f8660144980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.471+0000 7f86655e4640 1 -- 172.21.15.186:0/349093875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86600c2d00 msgr2=0x7f86601f18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.471+0000 7f86655e4640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86600c2d00 0x7f86601f18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.471+0000 7f86655e4640 1 -- 172.21.15.186:0/349093875 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86601f1de0 con 0x7f8660151990 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.471+0000 7f86655e4640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8660151990 0x7f86601eed60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8658007e60 tx=0x7f865800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.471+0000 7f865d7fa640 1 -- 172.21.15.186:0/349093875 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658014420 con 0x7f8660151990 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.471+0000 7f865d7fa640 1 -- 172.21.15.186:0/349093875 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8658019070 con 0x7f8660151990 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.471+0000 7f865d7fa640 1 -- 172.21.15.186:0/349093875 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658014820 con 0x7f8660151990 2024-03-28T15:02:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.471+0000 7f8667047640 1 -- 172.21.15.186:0/349093875 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86601ef2a0 con 0x7f8660151990 2024-03-28T15:02:03.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.471+0000 7f8667047640 1 -- 172.21.15.186:0/349093875 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86600cbb20 con 0x7f8660151990 2024-03-28T15:02:03.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.472+0000 7f8667047640 1 -- 172.21.15.186:0/349093875 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8660073670 con 0x7f8660151990 2024-03-28T15:02:03.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.472+0000 7f865d7fa640 1 -- 172.21.15.186:0/349093875 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f86580149c0 con 0x7f8660151990 2024-03-28T15:02:03.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.473+0000 7f865d7fa640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f864406c560 0x7f864406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:03.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.473+0000 7f865d7fa640 1 -- 172.21.15.186:0/349093875 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f865808be20 con 0x7f8660151990 2024-03-28T15:02:03.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.473+0000 7f8664de3640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f864406c560 0x7f864406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:03.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.473+0000 7f8664de3640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f864406c560 0x7f864406ea20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8654006fd0 tx=0x7f8654008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:03.477 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.477+0000 7f865d7fa640 1 -- 172.21.15.186:0/349093875 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8660073670 con 0x7f8660151990 2024-03-28T15:02:03.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.689+0000 7f8667047640 1 -- 172.21.15.186:0/349093875 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8660069500 con 0x7f8660151990 2024-03-28T15:02:03.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.690+0000 7f865d7fa640 1 -- 172.21.15.186:0/349093875 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7f8658059d60 con 0x7f8660151990 2024-03-28T15:02:03.690 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:03.690 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:02:03.690 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:02:03.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 -- 172.21.15.186:0/349093875 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f864406c560 msgr2=0x7f864406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:03.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f864406c560 0x7f864406ea20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8654006fd0 tx=0x7f8654008040 comp rx=0 tx=0).stop 2024-03-28T15:02:03.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 -- 172.21.15.186:0/349093875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8660151990 msgr2=0x7f86601eed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:03.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8660151990 0x7f86601eed60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8658007e60 tx=0x7f865800e4b0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 -- 172.21.15.186:0/349093875 shutdown_connections 2024-03-28T15:02:03.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f864406c560 0x7f864406ea20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8660151990 0x7f86601eed60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86600c2d00 0x7f86601f18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 --2- 172.21.15.186:0/349093875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86600b1e30 0x7f8660144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:03.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 -- 172.21.15.186:0/349093875 >> 172.21.15.186:0/349093875 conn(0x7f86600c8a20 msgr2=0x7f866006f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:03.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 -- 172.21.15.186:0/349093875 shutdown_connections 2024-03-28T15:02:03.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:03.692+0000 7f8667047640 1 -- 172.21.15.186:0/349093875 wait complete. 2024-03-28T15:02:03.703 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:02:05.704 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:05.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.817+0000 7f171aa6f640 1 Processor -- start 2024-03-28T15:02:05.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.817+0000 7f171aa6f640 1 -- start start 2024-03-28T15:02:05.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.817+0000 7f171aa6f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f17140caa70 0x7f17140c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:05.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.817+0000 7f171aa6f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f17140c6a00 0x7f17140cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:05.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.817+0000 7f171aa6f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17140c5380 0x7f17140cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:05.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.817+0000 7f171aa6f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f1714071a80 con 0x7f17140caa70 2024-03-28T15:02:05.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.817+0000 7f171aa6f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f17140c6030 con 0x7f17140c6a00 2024-03-28T15:02:05.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.817+0000 7f171aa6f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f17140c6f90 con 0x7f17140c5380 2024-03-28T15:02:05.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.818+0000 7f171880b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f17140caa70 0x7f17140c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:05.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.818+0000 7f171880b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f17140caa70 0x7f17140c64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36404/0 (socket says 172.21.15.186:36404) 2024-03-28T15:02:05.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.818+0000 7f171880b640 1 -- 172.21.15.186:0/2464770654 learned_addr learned my addr 172.21.15.186:0/2464770654 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:05.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.818+0000 7f171900c640 1 --2- 172.21.15.186:0/2464770654 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17140c5380 0x7f17140cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:05.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.818+0000 7f1713fff640 1 --2- 172.21.15.186:0/2464770654 >> v2:172.21.15.191:3300/0 conn(0x7f17140c6a00 0x7f17140cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:05.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.818+0000 7f171880b640 1 -- 172.21.15.186:0/2464770654 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17140c5380 msgr2=0x7f17140cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:05.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.818+0000 7f171880b640 1 --2- 172.21.15.186:0/2464770654 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17140c5380 0x7f17140cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:05.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.818+0000 7f171880b640 1 -- 172.21.15.186:0/2464770654 >> v2:172.21.15.191:3300/0 conn(0x7f17140c6a00 msgr2=0x7f17140cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:05.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.818+0000 7f171880b640 1 --2- 172.21.15.186:0/2464770654 >> v2:172.21.15.191:3300/0 conn(0x7f17140c6a00 0x7f17140cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:05.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.818+0000 7f171880b640 1 -- 172.21.15.186:0/2464770654 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17140cc970 con 0x7f17140caa70 2024-03-28T15:02:05.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f171880b640 1 --2- 172.21.15.186:0/2464770654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17140caa70 0x7f17140c64c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f171415efb0 tx=0x7f170c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6226c3e1b572b09b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f17137fe640 1 -- 172.21.15.186:0/2464770654 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170c02ca70 con 0x7f17140caa70 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f17137fe640 1 -- 172.21.15.186:0/2464770654 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f170c02cc10 con 0x7f17140caa70 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f17137fe640 1 -- 172.21.15.186:0/2464770654 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170c0343f0 con 0x7f17140caa70 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f171aa6f640 1 -- 172.21.15.186:0/2464770654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17140caa70 msgr2=0x7f17140c64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f171aa6f640 1 --2- 172.21.15.186:0/2464770654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17140caa70 0x7f17140c64c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f171415efb0 tx=0x7f170c02bed0 comp rx=0 tx=0).stop 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f171aa6f640 1 -- 172.21.15.186:0/2464770654 shutdown_connections 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f171aa6f640 1 --2- 172.21.15.186:0/2464770654 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17140c5380 0x7f17140cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f171aa6f640 1 --2- 172.21.15.186:0/2464770654 >> v2:172.21.15.191:3300/0 conn(0x7f17140c6a00 0x7f17140cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f171aa6f640 1 --2- 172.21.15.186:0/2464770654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17140caa70 0x7f17140c64c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f171aa6f640 1 -- 172.21.15.186:0/2464770654 >> 172.21.15.186:0/2464770654 conn(0x7f17140d00f0 msgr2=0x7f17140d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f171aa6f640 1 -- 172.21.15.186:0/2464770654 shutdown_connections 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.819+0000 7f171aa6f640 1 -- 172.21.15.186:0/2464770654 wait complete. 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.820+0000 7f171aa6f640 1 Processor -- start 2024-03-28T15:02:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.821+0000 7f171aa6f640 1 -- start start 2024-03-28T15:02:05.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.821+0000 7f171aa6f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17140c5380 0x7f171414be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:05.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.821+0000 7f171aa6f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17140c6a00 0x7f1714148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:05.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.821+0000 7f171aa6f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17140caa70 0x7f1714148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.821+0000 7f171aa6f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f171415b530 con 0x7f17140c6a00 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.821+0000 7f171aa6f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f17140cc690 con 0x7f17140caa70 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.821+0000 7f171aa6f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f17140c5cc0 con 0x7f17140c5380 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f1713fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17140c6a00 0x7f1714148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f1713fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17140c6a00 0x7f1714148a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36420/0 (socket says 172.21.15.186:36420) 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f1713fff640 1 -- 172.21.15.186:0/2069328842 learned_addr learned my addr 172.21.15.186:0/2069328842 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f171880b640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17140c5380 0x7f171414be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f171900c640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17140caa70 0x7f1714148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f1713fff640 1 -- 172.21.15.186:0/2069328842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17140c5380 msgr2=0x7f171414be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f1713fff640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17140c5380 0x7f171414be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f1713fff640 1 -- 172.21.15.186:0/2069328842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17140caa70 msgr2=0x7f1714148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f1713fff640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17140caa70 0x7f1714148f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f1713fff640 1 -- 172.21.15.186:0/2069328842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171414c6c0 con 0x7f17140c6a00 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f1713fff640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17140c6a00 0x7f1714148a40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f170400d800 tx=0x7f170400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f17117fa640 1 -- 172.21.15.186:0/2069328842 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704004020 con 0x7f17140c6a00 2024-03-28T15:02:05.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f17117fa640 1 -- 172.21.15.186:0/2069328842 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17040041c0 con 0x7f17140c6a00 2024-03-28T15:02:05.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f171aa6f640 1 -- 172.21.15.186:0/2069328842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171414c9a0 con 0x7f17140c6a00 2024-03-28T15:02:05.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f17117fa640 1 -- 172.21.15.186:0/2069328842 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704004390 con 0x7f17140c6a00 2024-03-28T15:02:05.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.822+0000 7f171aa6f640 1 -- 172.21.15.186:0/2069328842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17140c86a0 con 0x7f17140c6a00 2024-03-28T15:02:05.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.823+0000 7f171aa6f640 1 -- 172.21.15.186:0/2069328842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17140686d0 con 0x7f17140c6a00 2024-03-28T15:02:05.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.824+0000 7f17117fa640 1 -- 172.21.15.186:0/2069328842 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f170401b020 con 0x7f17140c6a00 2024-03-28T15:02:05.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.824+0000 7f17117fa640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f16f406c530 0x7f16f406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:05.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.824+0000 7f17117fa640 1 -- 172.21.15.186:0/2069328842 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f170408b2d0 con 0x7f17140c6a00 2024-03-28T15:02:05.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.824+0000 7f171880b640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f16f406c530 0x7f16f406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:05.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.824+0000 7f171880b640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f16f406c530 0x7f16f406e9f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f170c035000 tx=0x7f170c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:05.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:05.828+0000 7f17117fa640 1 -- 172.21.15.186:0/2069328842 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f1704059330 con 0x7f17140c6a00 2024-03-28T15:02:06.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.038+0000 7f171aa6f640 1 -- 172.21.15.186:0/2069328842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f171415a7f0 con 0x7f17140c6a00 2024-03-28T15:02:06.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.039+0000 7f17117fa640 1 -- 172.21.15.186:0/2069328842 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7f170405d630 con 0x7f17140c6a00 2024-03-28T15:02:06.039 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:06.040 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:02:06.040 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:02:06.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 -- 172.21.15.186:0/2069328842 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f16f406c530 msgr2=0x7f16f406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:06.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f16f406c530 0x7f16f406e9f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f170c035000 tx=0x7f170c030000 comp rx=0 tx=0).stop 2024-03-28T15:02:06.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 -- 172.21.15.186:0/2069328842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17140c6a00 msgr2=0x7f1714148a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:06.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17140c6a00 0x7f1714148a40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f170400d800 tx=0x7f170400dcc0 comp rx=0 tx=0).stop 2024-03-28T15:02:06.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 -- 172.21.15.186:0/2069328842 shutdown_connections 2024-03-28T15:02:06.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f16f406c530 0x7f16f406e9f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:06.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17140caa70 0x7f1714148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:06.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17140c6a00 0x7f1714148a40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:06.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 --2- 172.21.15.186:0/2069328842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17140c5380 0x7f171414be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:06.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 -- 172.21.15.186:0/2069328842 >> 172.21.15.186:0/2069328842 conn(0x7f17140d00f0 msgr2=0x7f171414ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:06.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 -- 172.21.15.186:0/2069328842 shutdown_connections 2024-03-28T15:02:06.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:06.041+0000 7f171aa6f640 1 -- 172.21.15.186:0/2069328842 wait complete. 2024-03-28T15:02:06.052 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:02:08.052 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:08.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.165+0000 7fcd1efe2640 1 Processor -- start 2024-03-28T15:02:08.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.165+0000 7fcd1efe2640 1 -- start start 2024-03-28T15:02:08.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1efe2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcd18150ed0 0x7fcd180c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:08.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1efe2640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcd180733f0 0x7fcd180b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:08.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1efe2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd180c1b40 con 0x7fcd181512b0 2024-03-28T15:02:08.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1efe2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fcd180c1e20 con 0x7fcd18150ed0 2024-03-28T15:02:08.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1efe2640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fcd180c18a0 con 0x7fcd180733f0 2024-03-28T15:02:08.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1cd7e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcd18150ed0 0x7fcd180c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1cd7e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcd18150ed0 0x7fcd180c0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36426/0 (socket says 172.21.15.186:36426) 2024-03-28T15:02:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1cd7e640 1 -- 172.21.15.186:0/215915247 learned_addr learned my addr 172.21.15.186:0/215915247 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1d57f640 1 --2- 172.21.15.186:0/215915247 >> v2:172.21.15.191:3300/0 conn(0x7fcd180733f0 0x7fcd180b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1cd7e640 1 -- 172.21.15.186:0/215915247 >> v2:172.21.15.191:3300/0 conn(0x7fcd180733f0 msgr2=0x7fcd180b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1cd7e640 1 --2- 172.21.15.186:0/215915247 >> v2:172.21.15.191:3300/0 conn(0x7fcd180733f0 0x7fcd180b1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.166+0000 7fcd1cd7e640 1 -- 172.21.15.186:0/215915247 >> v1:172.21.15.191:6789/0 conn(0x7fcd181512b0 legacy=0x7fcd18156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.167+0000 7fcd1cd7e640 1 -- 172.21.15.186:0/215915247 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd180c15c0 con 0x7fcd18150ed0 2024-03-28T15:02:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.167+0000 7fcd1d57f640 1 --2- 172.21.15.186:0/215915247 >> v2:172.21.15.191:3300/0 conn(0x7fcd180733f0 0x7fcd180b1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:02:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.167+0000 7fcd1cd7e640 1 --2- 172.21.15.186:0/215915247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd18150ed0 0x7fcd180c0f40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcd18067cd0 tx=0x7fcd0802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a36a53e2a8cec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:08.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.167+0000 7fcd177fe640 1 -- 172.21.15.186:0/215915247 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0802f850 con 0x7fcd18150ed0 2024-03-28T15:02:08.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.168+0000 7fcd177fe640 1 -- 172.21.15.186:0/215915247 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0802f9f0 con 0x7fcd18150ed0 2024-03-28T15:02:08.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.168+0000 7fcd177fe640 1 -- 172.21.15.186:0/215915247 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0802fdc0 con 0x7fcd18150ed0 2024-03-28T15:02:08.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.168+0000 7fcd1efe2640 1 -- 172.21.15.186:0/215915247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd18150ed0 msgr2=0x7fcd180c0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:08.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.168+0000 7fcd1efe2640 1 --2- 172.21.15.186:0/215915247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd18150ed0 0x7fcd180c0f40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcd18067cd0 tx=0x7fcd0802ba30 comp rx=0 tx=0).stop 2024-03-28T15:02:08.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.168+0000 7fcd1efe2640 1 -- 172.21.15.186:0/215915247 shutdown_connections 2024-03-28T15:02:08.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.168+0000 7fcd1efe2640 1 --2- 172.21.15.186:0/215915247 >> v2:172.21.15.191:3300/0 conn(0x7fcd180733f0 0x7fcd180b1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:08.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.168+0000 7fcd1efe2640 1 --2- 172.21.15.186:0/215915247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd18150ed0 0x7fcd180c0f40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:08.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.168+0000 7fcd1efe2640 1 -- 172.21.15.186:0/215915247 >> 172.21.15.186:0/215915247 conn(0x7fcd180c8a20 msgr2=0x7fcd180c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:08.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.168+0000 7fcd1efe2640 1 -- 172.21.15.186:0/215915247 shutdown_connections 2024-03-28T15:02:08.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.168+0000 7fcd1efe2640 1 -- 172.21.15.186:0/215915247 wait complete. 2024-03-28T15:02:08.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.169+0000 7fcd1efe2640 1 Processor -- start 2024-03-28T15:02:08.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.169+0000 7fcd1efe2640 1 -- start start 2024-03-28T15:02:08.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.170+0000 7fcd1efe2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd180733f0 0x7fcd18142e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:08.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.170+0000 7fcd1efe2640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd18150ed0 0x7fcd1814fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:08.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.170+0000 7fcd1efe2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd181512b0 0x7fcd1813eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:08.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.170+0000 7fcd1efe2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1806e5b0 con 0x7fcd180733f0 2024-03-28T15:02:08.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.170+0000 7fcd1efe2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcd18072f60 con 0x7fcd181512b0 2024-03-28T15:02:08.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.170+0000 7fcd1efe2640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fcd180712b0 con 0x7fcd18150ed0 2024-03-28T15:02:08.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.170+0000 7fcd1cd7e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd180733f0 0x7fcd18142e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:08.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.170+0000 7fcd1cd7e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd180733f0 0x7fcd18142e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36436/0 (socket says 172.21.15.186:36436) 2024-03-28T15:02:08.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.170+0000 7fcd1cd7e640 1 -- 172.21.15.186:0/424730757 learned_addr learned my addr 172.21.15.186:0/424730757 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:08.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.170+0000 7fcd1d57f640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd181512b0 0x7fcd1813eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:08.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.170+0000 7fcd17fff640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd18150ed0 0x7fcd1814fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:08.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.171+0000 7fcd1cd7e640 1 -- 172.21.15.186:0/424730757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd18150ed0 msgr2=0x7fcd1814fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:08.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.171+0000 7fcd1cd7e640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd18150ed0 0x7fcd1814fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:08.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.171+0000 7fcd1cd7e640 1 -- 172.21.15.186:0/424730757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd181512b0 msgr2=0x7fcd1813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:08.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.171+0000 7fcd1cd7e640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd181512b0 0x7fcd1813eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:08.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.171+0000 7fcd1cd7e640 1 -- 172.21.15.186:0/424730757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd181f3dc0 con 0x7fcd180733f0 2024-03-28T15:02:08.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.171+0000 7fcd1cd7e640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd180733f0 0x7fcd18142e80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcd08000c00 tx=0x7fcd08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:08.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.171+0000 7fcd157fa640 1 -- 172.21.15.186:0/424730757 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08047070 con 0x7fcd180733f0 2024-03-28T15:02:08.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.171+0000 7fcd157fa640 1 -- 172.21.15.186:0/424730757 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd08037cb0 con 0x7fcd180733f0 2024-03-28T15:02:08.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.171+0000 7fcd157fa640 1 -- 172.21.15.186:0/424730757 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08037e60 con 0x7fcd180733f0 2024-03-28T15:02:08.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.171+0000 7fcd1efe2640 1 -- 172.21.15.186:0/424730757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd181f4060 con 0x7fcd180733f0 2024-03-28T15:02:08.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.171+0000 7fcd1efe2640 1 -- 172.21.15.186:0/424730757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd181f4620 con 0x7fcd180733f0 2024-03-28T15:02:08.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.172+0000 7fcd1efe2640 1 -- 172.21.15.186:0/424730757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd1814f430 con 0x7fcd180733f0 2024-03-28T15:02:08.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.173+0000 7fcd157fa640 1 -- 172.21.15.186:0/424730757 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fcd0804a070 con 0x7fcd180733f0 2024-03-28T15:02:08.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.173+0000 7fcd157fa640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0406c530 0x7fcd0406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:08.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.173+0000 7fcd157fa640 1 -- 172.21.15.186:0/424730757 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcd080adaa0 con 0x7fcd180733f0 2024-03-28T15:02:08.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.173+0000 7fcd17fff640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0406c530 0x7fcd0406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:08.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.173+0000 7fcd17fff640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0406c530 0x7fcd0406e9f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcd18163140 tx=0x7fcd0c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:08.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.178+0000 7fcd157fa640 1 -- 172.21.15.186:0/424730757 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fcd1814f430 con 0x7fcd180733f0 2024-03-28T15:02:08.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.389+0000 7fcd1efe2640 1 -- 172.21.15.186:0/424730757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd18069500 con 0x7fcd180733f0 2024-03-28T15:02:08.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.390+0000 7fcd157fa640 1 -- 172.21.15.186:0/424730757 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7fcd08033030 con 0x7fcd180733f0 2024-03-28T15:02:08.390 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:08.391 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:02:08.391 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:02:08.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 -- 172.21.15.186:0/424730757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0406c530 msgr2=0x7fcd0406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:08.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0406c530 0x7fcd0406e9f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcd18163140 tx=0x7fcd0c009040 comp rx=0 tx=0).stop 2024-03-28T15:02:08.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 -- 172.21.15.186:0/424730757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd180733f0 msgr2=0x7fcd18142e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:08.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd180733f0 0x7fcd18142e80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcd08000c00 tx=0x7fcd08004730 comp rx=0 tx=0).stop 2024-03-28T15:02:08.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 -- 172.21.15.186:0/424730757 shutdown_connections 2024-03-28T15:02:08.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0406c530 0x7fcd0406e9f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:08.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd181512b0 0x7fcd1813eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:08.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd18150ed0 0x7fcd1814fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:08.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 --2- 172.21.15.186:0/424730757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd180733f0 0x7fcd18142e80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:08.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 -- 172.21.15.186:0/424730757 >> 172.21.15.186:0/424730757 conn(0x7fcd180c8a20 msgr2=0x7fcd180c7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:08.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 -- 172.21.15.186:0/424730757 shutdown_connections 2024-03-28T15:02:08.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:08.392+0000 7fcd1efe2640 1 -- 172.21.15.186:0/424730757 wait complete. 2024-03-28T15:02:08.404 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:02:10.404 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:10.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.516+0000 7fea31a33640 1 Processor -- start 2024-03-28T15:02:10.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.516+0000 7fea31a33640 1 -- start start 2024-03-28T15:02:10.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea31a33640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fea2c154bd0 0x7fea2c15a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:10.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea31a33640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fea2c0c53c0 0x7fea2c0c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:10.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea31a33640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c0c57e0 0x7fea2c0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:10.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea31a33640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fea2c0c5ca0 con 0x7fea2c154bd0 2024-03-28T15:02:10.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea31a33640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fea2c15b080 con 0x7fea2c0c53c0 2024-03-28T15:02:10.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea31a33640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fea2c15b360 con 0x7fea2c0c57e0 2024-03-28T15:02:10.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea2affd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fea2c154bd0 0x7fea2c15a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:10.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea2affd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fea2c154bd0 0x7fea2c15a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43804/0 (socket says 172.21.15.186:43804) 2024-03-28T15:02:10.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea2affd640 1 -- 172.21.15.186:0/17699286 learned_addr learned my addr 172.21.15.186:0/17699286 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:10.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea2b7fe640 1 --2- 172.21.15.186:0/17699286 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c0c57e0 0x7fea2c0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:10.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea2a7fc640 1 --2- 172.21.15.186:0/17699286 >> v2:172.21.15.191:3300/0 conn(0x7fea2c0c53c0 0x7fea2c0c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:10.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea2affd640 1 -- 172.21.15.186:0/17699286 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c0c57e0 msgr2=0x7fea2c0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:10.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea2affd640 1 --2- 172.21.15.186:0/17699286 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c0c57e0 0x7fea2c0cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea2affd640 1 -- 172.21.15.186:0/17699286 >> v2:172.21.15.191:3300/0 conn(0x7fea2c0c53c0 msgr2=0x7fea2c0c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:10.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea2affd640 1 --2- 172.21.15.186:0/17699286 >> v2:172.21.15.191:3300/0 conn(0x7fea2c0c53c0 0x7fea2c0c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea2affd640 1 -- 172.21.15.186:0/17699286 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2c15b640 con 0x7fea2c154bd0 2024-03-28T15:02:10.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.517+0000 7fea2a7fc640 1 --2- 172.21.15.186:0/17699286 >> v2:172.21.15.191:3300/0 conn(0x7fea2c0c53c0 0x7fea2c0c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:02:10.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.518+0000 7fea2affd640 1 --2- 172.21.15.186:0/17699286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea2c154bd0 0x7fea2c15a7d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fea2c08d6b0 tx=0x7fea1c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241750f1dd29a2a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:10.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.518+0000 7fea29ffb640 1 -- 172.21.15.186:0/17699286 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c02cb00 con 0x7fea2c154bd0 2024-03-28T15:02:10.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.518+0000 7fea29ffb640 1 -- 172.21.15.186:0/17699286 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea1c02cca0 con 0x7fea2c154bd0 2024-03-28T15:02:10.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.518+0000 7fea29ffb640 1 -- 172.21.15.186:0/17699286 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c034610 con 0x7fea2c154bd0 2024-03-28T15:02:10.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.519+0000 7fea31a33640 1 -- 172.21.15.186:0/17699286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea2c154bd0 msgr2=0x7fea2c15a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:10.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.519+0000 7fea31a33640 1 --2- 172.21.15.186:0/17699286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea2c154bd0 0x7fea2c15a7d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fea2c08d6b0 tx=0x7fea1c02bf60 comp rx=0 tx=0).stop 2024-03-28T15:02:10.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.519+0000 7fea31a33640 1 -- 172.21.15.186:0/17699286 shutdown_connections 2024-03-28T15:02:10.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.519+0000 7fea31a33640 1 --2- 172.21.15.186:0/17699286 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c0c57e0 0x7fea2c0cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.519+0000 7fea31a33640 1 --2- 172.21.15.186:0/17699286 >> v2:172.21.15.191:3300/0 conn(0x7fea2c0c53c0 0x7fea2c0c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.519+0000 7fea31a33640 1 --2- 172.21.15.186:0/17699286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea2c154bd0 0x7fea2c15a7d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.519+0000 7fea31a33640 1 -- 172.21.15.186:0/17699286 >> 172.21.15.186:0/17699286 conn(0x7fea2c0731f0 msgr2=0x7fea2c073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:10.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.519+0000 7fea31a33640 1 -- 172.21.15.186:0/17699286 shutdown_connections 2024-03-28T15:02:10.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.519+0000 7fea31a33640 1 -- 172.21.15.186:0/17699286 wait complete. 2024-03-28T15:02:10.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.520+0000 7fea31a33640 1 Processor -- start 2024-03-28T15:02:10.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.520+0000 7fea31a33640 1 -- start start 2024-03-28T15:02:10.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea31a33640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c0c53c0 0x7fea2c15c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:10.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea31a33640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea2c0c57e0 0x7fea2c15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:10.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea31a33640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c154bd0 0x7fea2c14a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:10.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea31a33640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c0cc390 con 0x7fea2c0c57e0 2024-03-28T15:02:10.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea31a33640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c0c61d0 con 0x7fea2c154bd0 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea31a33640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fea2c15ad10 con 0x7fea2c0c53c0 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea2a7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea2c0c57e0 0x7fea2c15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea2a7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea2c0c57e0 0x7fea2c15cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43812/0 (socket says 172.21.15.186:43812) 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea2a7fc640 1 -- 172.21.15.186:0/2458007249 learned_addr learned my addr 172.21.15.186:0/2458007249 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea2b7fe640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c154bd0 0x7fea2c14a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea2affd640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c0c53c0 0x7fea2c15c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea2a7fc640 1 -- 172.21.15.186:0/2458007249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c0c53c0 msgr2=0x7fea2c15c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea2a7fc640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c0c53c0 0x7fea2c15c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea2a7fc640 1 -- 172.21.15.186:0/2458007249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c154bd0 msgr2=0x7fea2c14a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea2a7fc640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c154bd0 0x7fea2c14a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.521+0000 7fea2a7fc640 1 -- 172.21.15.186:0/2458007249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2c0b2e20 con 0x7fea2c0c57e0 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.522+0000 7fea2a7fc640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea2c0c57e0 0x7fea2c15cea0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fea2000c7f0 tx=0x7fea2000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.522+0000 7fea13fff640 1 -- 172.21.15.186:0/2458007249 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea20015070 con 0x7fea2c0c57e0 2024-03-28T15:02:10.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.522+0000 7fea13fff640 1 -- 172.21.15.186:0/2458007249 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea20007a30 con 0x7fea2c0c57e0 2024-03-28T15:02:10.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.522+0000 7fea31a33640 1 -- 172.21.15.186:0/2458007249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2c153ce0 con 0x7fea2c0c57e0 2024-03-28T15:02:10.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.522+0000 7fea13fff640 1 -- 172.21.15.186:0/2458007249 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea20007e00 con 0x7fea2c0c57e0 2024-03-28T15:02:10.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.522+0000 7fea31a33640 1 -- 172.21.15.186:0/2458007249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea2c1542a0 con 0x7fea2c0c57e0 2024-03-28T15:02:10.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.523+0000 7fea31a33640 1 -- 172.21.15.186:0/2458007249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea2c069500 con 0x7fea2c0c57e0 2024-03-28T15:02:10.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.523+0000 7fea13fff640 1 -- 172.21.15.186:0/2458007249 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fea20018070 con 0x7fea2c0c57e0 2024-03-28T15:02:10.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.524+0000 7fea13fff640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fea1806c530 0x7fea1806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:10.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.524+0000 7fea13fff640 1 -- 172.21.15.186:0/2458007249 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fea2008b600 con 0x7fea2c0c57e0 2024-03-28T15:02:10.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.524+0000 7fea2affd640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fea1806c530 0x7fea1806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:10.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.524+0000 7fea2affd640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fea1806c530 0x7fea1806e9f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fea1c035000 tx=0x7fea1c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:10.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.528+0000 7fea13fff640 1 -- 172.21.15.186:0/2458007249 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fea20059c20 con 0x7fea2c0c57e0 2024-03-28T15:02:10.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.740+0000 7fea31a33640 1 -- 172.21.15.186:0/2458007249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea2c0b3280 con 0x7fea2c0c57e0 2024-03-28T15:02:10.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.741+0000 7fea13fff640 1 -- 172.21.15.186:0/2458007249 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6839 (secure 0 0 0) 0x7fea20059520 con 0x7fea2c0c57e0 2024-03-28T15:02:10.741 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:10.741 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:01:38.125799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/2813406930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":2813406930},{"type":"v1","addr":"172.21.15.191:6837","nonce":2813406930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4308,"qdb_cluster":[4308]},"id":1}]} 2024-03-28T15:02:10.741 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 8 2024-03-28T15:02:10.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 -- 172.21.15.186:0/2458007249 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fea1806c530 msgr2=0x7fea1806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:10.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fea1806c530 0x7fea1806e9f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fea1c035000 tx=0x7fea1c030000 comp rx=0 tx=0).stop 2024-03-28T15:02:10.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 -- 172.21.15.186:0/2458007249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea2c0c57e0 msgr2=0x7fea2c15cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:10.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea2c0c57e0 0x7fea2c15cea0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fea2000c7f0 tx=0x7fea2000ccb0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 -- 172.21.15.186:0/2458007249 shutdown_connections 2024-03-28T15:02:10.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fea1806c530 0x7fea1806e9f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c154bd0 0x7fea2c14a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea2c0c57e0 0x7fea2c15cea0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 --2- 172.21.15.186:0/2458007249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c0c53c0 0x7fea2c15c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:10.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 -- 172.21.15.186:0/2458007249 >> 172.21.15.186:0/2458007249 conn(0x7fea2c0731f0 msgr2=0x7fea2c0cd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:10.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 -- 172.21.15.186:0/2458007249 shutdown_connections 2024-03-28T15:02:10.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:10.743+0000 7fea31a33640 1 -- 172.21.15.186:0/2458007249 wait complete. 2024-03-28T15:02:10.754 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:02:12.755 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:12.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.866+0000 7f324c15e640 1 Processor -- start 2024-03-28T15:02:12.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.866+0000 7f324c15e640 1 -- start start 2024-03-28T15:02:12.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.867+0000 7f324c15e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3244154bd0 0x7f324415a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:12.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.867+0000 7f324c15e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32440c53c0 0x7f32440c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:12.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.867+0000 7f324c15e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f324415b080 con 0x7f32440c57e0 2024-03-28T15:02:12.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.867+0000 7f324c15e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f32440c5ca0 con 0x7f3244154bd0 2024-03-28T15:02:12.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.867+0000 7f324c15e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f324415b360 con 0x7f32440c53c0 2024-03-28T15:02:12.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.867+0000 7f324a6fb640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f32440c57e0 0x7f32440cb040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:60350/0 (socket says 172.21.15.186:60350) 2024-03-28T15:02:12.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.867+0000 7f324a6fb640 1 -- 172.21.15.186:0/3522555846 learned_addr learned my addr 172.21.15.186:0/3522555846 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:02:12.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.867+0000 7f3249efa640 1 --2- 172.21.15.186:0/3522555846 >> v2:172.21.15.191:3300/0 conn(0x7f3244154bd0 0x7f324415a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:12.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.867+0000 7f32496f9640 1 --2- 172.21.15.186:0/3522555846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32440c53c0 0x7f32440c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:12.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.868+0000 7f3248ef8640 1 -- 172.21.15.186:0/3522555846 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2790812861 0 0) 0x7f324415b080 con 0x7f32440c57e0 2024-03-28T15:02:12.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.868+0000 7f3248ef8640 1 -- 172.21.15.186:0/3522555846 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3230003520 con 0x7f32440c57e0 2024-03-28T15:02:12.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.868+0000 7f3249efa640 1 -- 172.21.15.186:0/3522555846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32440c53c0 msgr2=0x7f32440c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:12.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.868+0000 7f3249efa640 1 --2- 172.21.15.186:0/3522555846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32440c53c0 0x7f32440c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:12.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.868+0000 7f3249efa640 1 -- 172.21.15.186:0/3522555846 >> v1:172.21.15.186:6789/0 conn(0x7f32440c57e0 legacy=0x7f32440cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:12.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.868+0000 7f3249efa640 1 -- 172.21.15.186:0/3522555846 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324415b640 con 0x7f3244154bd0 2024-03-28T15:02:12.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.868+0000 7f32496f9640 1 --2- 172.21.15.186:0/3522555846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32440c53c0 0x7f32440c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:02:12.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.869+0000 7f3249efa640 1 --2- 172.21.15.186:0/3522555846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3244154bd0 0x7f324415a7d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f32400097a0 tx=0x7f324002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0ec49d1ee17c74 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:12.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.869+0000 7f3248ef8640 1 -- 172.21.15.186:0/3522555846 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240034070 con 0x7f3244154bd0 2024-03-28T15:02:12.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.869+0000 7f3248ef8640 1 -- 172.21.15.186:0/3522555846 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3240030610 con 0x7f3244154bd0 2024-03-28T15:02:12.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.869+0000 7f3248ef8640 1 -- 172.21.15.186:0/3522555846 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32400309c0 con 0x7f3244154bd0 2024-03-28T15:02:12.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.869+0000 7f324c15e640 1 -- 172.21.15.186:0/3522555846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3244154bd0 msgr2=0x7f324415a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:12.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.869+0000 7f324c15e640 1 --2- 172.21.15.186:0/3522555846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3244154bd0 0x7f324415a7d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f32400097a0 tx=0x7f324002b960 comp rx=0 tx=0).stop 2024-03-28T15:02:12.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.869+0000 7f324c15e640 1 -- 172.21.15.186:0/3522555846 shutdown_connections 2024-03-28T15:02:12.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.869+0000 7f324c15e640 1 --2- 172.21.15.186:0/3522555846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32440c53c0 0x7f32440c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:12.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.869+0000 7f324c15e640 1 --2- 172.21.15.186:0/3522555846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3244154bd0 0x7f324415a7d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:12.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.869+0000 7f324c15e640 1 -- 172.21.15.186:0/3522555846 >> 172.21.15.186:0/3522555846 conn(0x7f32440731f0 msgr2=0x7f3244073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:12.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.870+0000 7f324c15e640 1 -- 172.21.15.186:0/3522555846 shutdown_connections 2024-03-28T15:02:12.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.870+0000 7f324c15e640 1 -- 172.21.15.186:0/3522555846 wait complete. 2024-03-28T15:02:12.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.871+0000 7f324c15e640 1 Processor -- start 2024-03-28T15:02:12.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.871+0000 7f324c15e640 1 -- start start 2024-03-28T15:02:12.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.871+0000 7f324c15e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32440c53c0 0x7f324415e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:12.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.871+0000 7f324c15e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32440c57e0 0x7f32441e5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:12.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.871+0000 7f324c15e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3244154bd0 0x7f32441f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:12.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.871+0000 7f324c15e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f32440cc390 con 0x7f32440c57e0 2024-03-28T15:02:12.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.871+0000 7f324c15e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f32440c61d0 con 0x7f3244154bd0 2024-03-28T15:02:12.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.871+0000 7f324c15e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f324415ad10 con 0x7f32440c53c0 2024-03-28T15:02:12.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.871+0000 7f32496f9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32440c57e0 0x7f32441e5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:12.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f32496f9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32440c57e0 0x7f32441e5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43822/0 (socket says 172.21.15.186:43822) 2024-03-28T15:02:12.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f32496f9640 1 -- 172.21.15.186:0/1406965365 learned_addr learned my addr 172.21.15.186:0/1406965365 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:12.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f3249efa640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32440c53c0 0x7f324415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:12.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f324a6fb640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3244154bd0 0x7f32441f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:12.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f32496f9640 1 -- 172.21.15.186:0/1406965365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32440c53c0 msgr2=0x7f324415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:12.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f32496f9640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32440c53c0 0x7f324415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:12.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f32496f9640 1 -- 172.21.15.186:0/1406965365 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3244154bd0 msgr2=0x7f32441f3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:12.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f32496f9640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3244154bd0 0x7f32441f3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:12.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f32496f9640 1 -- 172.21.15.186:0/1406965365 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32441e5fd0 con 0x7f32440c57e0 2024-03-28T15:02:12.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f324a6fb640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3244154bd0 0x7f32441f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:02:12.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f32496f9640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32440c57e0 0x7f32441e5a90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f323400ebe0 tx=0x7f323400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:12.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f323a7fc640 1 -- 172.21.15.186:0/1406965365 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3234017070 con 0x7f32440c57e0 2024-03-28T15:02:12.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f323a7fc640 1 -- 172.21.15.186:0/1406965365 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f323400cc30 con 0x7f32440c57e0 2024-03-28T15:02:12.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f323a7fc640 1 -- 172.21.15.186:0/1406965365 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3234013470 con 0x7f32440c57e0 2024-03-28T15:02:12.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f324c15e640 1 -- 172.21.15.186:0/1406965365 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32441578d0 con 0x7f32440c57e0 2024-03-28T15:02:12.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.872+0000 7f324c15e640 1 -- 172.21.15.186:0/1406965365 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32441e40f0 con 0x7f32440c57e0 2024-03-28T15:02:12.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.873+0000 7f324c15e640 1 -- 172.21.15.186:0/1406965365 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32440ca780 con 0x7f32440c57e0 2024-03-28T15:02:12.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.874+0000 7f323a7fc640 1 -- 172.21.15.186:0/1406965365 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f323401a070 con 0x7f32440c57e0 2024-03-28T15:02:12.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.874+0000 7f323a7fc640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322806c530 0x7f322806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:12.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.874+0000 7f323a7fc640 1 -- 172.21.15.186:0/1406965365 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5911+0+0 (secure 0 0 0) 0x7f323408b630 con 0x7f32440c57e0 2024-03-28T15:02:12.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.874+0000 7f3249efa640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322806c530 0x7f322806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:12.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.874+0000 7f3249efa640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322806c530 0x7f322806e9f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3240036000 tx=0x7f324003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:12.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:12.878+0000 7f323a7fc640 1 -- 172.21.15.186:0/1406965365 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f32440ca780 con 0x7f32440c57e0 2024-03-28T15:02:13.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.093+0000 7f324c15e640 1 -- 172.21.15.186:0/1406965365 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3244072790 con 0x7f32440c57e0 2024-03-28T15:02:13.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.094+0000 7f323a7fc640 1 -- 172.21.15.186:0/1406965365 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6006 (secure 0 0 0) 0x7f3244072790 con 0x7f32440c57e0 2024-03-28T15:02:13.094 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:13.094 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:11.169599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:02:13.095 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 9 2024-03-28T15:02:13.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 -- 172.21.15.186:0/1406965365 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322806c530 msgr2=0x7f322806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:13.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322806c530 0x7f322806e9f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3240036000 tx=0x7f324003b040 comp rx=0 tx=0).stop 2024-03-28T15:02:13.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 -- 172.21.15.186:0/1406965365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32440c57e0 msgr2=0x7f32441e5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:13.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32440c57e0 0x7f32441e5a90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f323400ebe0 tx=0x7f323400c520 comp rx=0 tx=0).stop 2024-03-28T15:02:13.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 -- 172.21.15.186:0/1406965365 shutdown_connections 2024-03-28T15:02:13.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322806c530 0x7f322806e9f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3244154bd0 0x7f32441f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32440c57e0 0x7f32441e5a90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 --2- 172.21.15.186:0/1406965365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32440c53c0 0x7f324415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 -- 172.21.15.186:0/1406965365 >> 172.21.15.186:0/1406965365 conn(0x7f32440731f0 msgr2=0x7f32440cf550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:13.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 -- 172.21.15.186:0/1406965365 shutdown_connections 2024-03-28T15:02:13.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.096+0000 7f324c15e640 1 -- 172.21.15.186:0/1406965365 wait complete. 2024-03-28T15:02:13.107 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-03-28T15:02:13.108 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-28T15:02:13.108 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:13.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.209+0000 7fa2ec62a640 1 Processor -- start 2024-03-28T15:02:13.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.209+0000 7fa2ec62a640 1 -- start start 2024-03-28T15:02:13.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.209+0000 7fa2ec62a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2e4151990 0x7fa2e40c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:13.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.209+0000 7fa2ec62a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa2e40b1e30 0x7fa2e40c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:13.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.209+0000 7fa2ec62a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e406caa0 con 0x7fa2e40c2d00 2024-03-28T15:02:13.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.209+0000 7fa2ec62a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa2e40c1da0 con 0x7fa2e40b1e30 2024-03-28T15:02:13.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.209+0000 7fa2ec62a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e406cd80 con 0x7fa2e4151990 2024-03-28T15:02:13.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.210+0000 7fa2e9bc5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa2e40b1e30 0x7fa2e40c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:13.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.210+0000 7fa2e9bc5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa2e40b1e30 0x7fa2e40c2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43824/0 (socket says 172.21.15.186:43824) 2024-03-28T15:02:13.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.210+0000 7fa2e9bc5640 1 -- 172.21.15.186:0/2846450655 learned_addr learned my addr 172.21.15.186:0/2846450655 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:13.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.210+0000 7fa2ea3c6640 1 --2- 172.21.15.186:0/2846450655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2e4151990 0x7fa2e40c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:13.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.210+0000 7fa2e9bc5640 1 -- 172.21.15.186:0/2846450655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2e4151990 msgr2=0x7fa2e40c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:13.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.210+0000 7fa2e9bc5640 1 --2- 172.21.15.186:0/2846450655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2e4151990 0x7fa2e40c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.210+0000 7fa2e9bc5640 1 -- 172.21.15.186:0/2846450655 >> v1:172.21.15.191:6789/0 conn(0x7fa2e40c2d00 legacy=0x7fa2e40712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:13.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.210+0000 7fa2e9bc5640 1 -- 172.21.15.186:0/2846450655 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e40c1340 con 0x7fa2e40b1e30 2024-03-28T15:02:13.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.211+0000 7fa2e9bc5640 1 --2- 172.21.15.186:0/2846450655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2e40b1e30 0x7fa2e40c2770 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800bb10 tx=0x7fa2d802e1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c6167df6c5afc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:13.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.211+0000 7fa2e93c4640 1 -- 172.21.15.186:0/2846450655 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d803b070 con 0x7fa2e40b1e30 2024-03-28T15:02:13.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.211+0000 7fa2e93c4640 1 -- 172.21.15.186:0/2846450655 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d8036440 con 0x7fa2e40b1e30 2024-03-28T15:02:13.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.211+0000 7fa2e93c4640 1 -- 172.21.15.186:0/2846450655 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8036810 con 0x7fa2e40b1e30 2024-03-28T15:02:13.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.211+0000 7fa2ec62a640 1 -- 172.21.15.186:0/2846450655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2e40b1e30 msgr2=0x7fa2e40c2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:13.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.211+0000 7fa2ec62a640 1 --2- 172.21.15.186:0/2846450655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2e40b1e30 0x7fa2e40c2770 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800bb10 tx=0x7fa2d802e1f0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.211+0000 7fa2ec62a640 1 -- 172.21.15.186:0/2846450655 shutdown_connections 2024-03-28T15:02:13.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.211+0000 7fa2ec62a640 1 --2- 172.21.15.186:0/2846450655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2e40b1e30 0x7fa2e40c2770 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.211+0000 7fa2ec62a640 1 --2- 172.21.15.186:0/2846450655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2e4151990 0x7fa2e40c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.212+0000 7fa2ec62a640 1 -- 172.21.15.186:0/2846450655 >> 172.21.15.186:0/2846450655 conn(0x7fa2e40c8a20 msgr2=0x7fa2e40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:13.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.212+0000 7fa2ec62a640 1 -- 172.21.15.186:0/2846450655 shutdown_connections 2024-03-28T15:02:13.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.212+0000 7fa2ec62a640 1 -- 172.21.15.186:0/2846450655 wait complete. 2024-03-28T15:02:13.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.213+0000 7fa2ec62a640 1 Processor -- start 2024-03-28T15:02:13.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.213+0000 7fa2ec62a640 1 -- start start 2024-03-28T15:02:13.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.213+0000 7fa2ec62a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2e40b1e30 0x7fa2e41ec230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:13.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.213+0000 7fa2ec62a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2e40c2d00 0x7fa2e41463c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:13.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.213+0000 7fa2ec62a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2e4151990 0x7fa2e4147830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:13.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.213+0000 7fa2ec62a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e4073dd0 con 0x7fa2e4151990 2024-03-28T15:02:13.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.213+0000 7fa2ec62a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e40c1900 con 0x7fa2e40b1e30 2024-03-28T15:02:13.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.213+0000 7fa2ec62a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e4073f50 con 0x7fa2e40c2d00 2024-03-28T15:02:13.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.213+0000 7fa2eabc7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2e4151990 0x7fa2e4147830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.213+0000 7fa2eabc7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2e4151990 0x7fa2e4147830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43832/0 (socket says 172.21.15.186:43832) 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.213+0000 7fa2eabc7640 1 -- 172.21.15.186:0/416296612 learned_addr learned my addr 172.21.15.186:0/416296612 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2e9bc5640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2e40c2d00 0x7fa2e41463c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2ea3c6640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2e40b1e30 0x7fa2e41ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2eabc7640 1 -- 172.21.15.186:0/416296612 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2e40c2d00 msgr2=0x7fa2e41463c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2eabc7640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2e40c2d00 0x7fa2e41463c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2eabc7640 1 -- 172.21.15.186:0/416296612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2e40b1e30 msgr2=0x7fa2e41ec230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2eabc7640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2e40b1e30 0x7fa2e41ec230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2eabc7640 1 -- 172.21.15.186:0/416296612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e41ec850 con 0x7fa2e4151990 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2eabc7640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2e4151990 0x7fa2e4147830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400c3c0 tx=0x7fa2d400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2e2ffd640 1 -- 172.21.15.186:0/416296612 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4017070 con 0x7fa2e4151990 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2e2ffd640 1 -- 172.21.15.186:0/416296612 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d40052d0 con 0x7fa2e4151990 2024-03-28T15:02:13.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2ec62a640 1 -- 172.21.15.186:0/416296612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e41ecb30 con 0x7fa2e4151990 2024-03-28T15:02:13.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2e2ffd640 1 -- 172.21.15.186:0/416296612 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4003810 con 0x7fa2e4151990 2024-03-28T15:02:13.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.214+0000 7fa2ec62a640 1 -- 172.21.15.186:0/416296612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e41ed0f0 con 0x7fa2e4151990 2024-03-28T15:02:13.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.215+0000 7fa2ec62a640 1 -- 172.21.15.186:0/416296612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e414ba10 con 0x7fa2e4151990 2024-03-28T15:02:13.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.215+0000 7fa2e2ffd640 1 -- 172.21.15.186:0/416296612 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa2d401a070 con 0x7fa2e4151990 2024-03-28T15:02:13.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.216+0000 7fa2e2ffd640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2d006c560 0x7fa2d006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:13.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.216+0000 7fa2e2ffd640 1 -- 172.21.15.186:0/416296612 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5911+0+0 (secure 0 0 0) 0x7fa2d408ab10 con 0x7fa2e4151990 2024-03-28T15:02:13.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.216+0000 7fa2ea3c6640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2d006c560 0x7fa2d006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:13.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.216+0000 7fa2ea3c6640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2d006c560 0x7fa2d006ea20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009f60 tx=0x7fa2dc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:13.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.220+0000 7fa2e2ffd640 1 -- 172.21.15.186:0/416296612 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa2e414ba10 con 0x7fa2e4151990 2024-03-28T15:02:13.432 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.432+0000 7fa2ec62a640 1 -- 172.21.15.186:0/416296612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2e4069500 con 0x7fa2e4151990 2024-03-28T15:02:13.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.432+0000 7fa2e2ffd640 1 -- 172.21.15.186:0/416296612 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6006 (secure 0 0 0) 0x7fa2d4058fd0 con 0x7fa2e4151990 2024-03-28T15:02:13.433 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:13.433 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:11.169599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:02:13.433 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 9 2024-03-28T15:02:13.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 -- 172.21.15.186:0/416296612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2d006c560 msgr2=0x7fa2d006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:13.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2d006c560 0x7fa2d006ea20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009f60 tx=0x7fa2dc00b040 comp rx=0 tx=0).stop 2024-03-28T15:02:13.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 -- 172.21.15.186:0/416296612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2e4151990 msgr2=0x7fa2e4147830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:13.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2e4151990 0x7fa2e4147830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400c3c0 tx=0x7fa2d400c880 comp rx=0 tx=0).stop 2024-03-28T15:02:13.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 -- 172.21.15.186:0/416296612 shutdown_connections 2024-03-28T15:02:13.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2d006c560 0x7fa2d006ea20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2e4151990 0x7fa2e4147830 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2e40c2d00 0x7fa2e41463c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 --2- 172.21.15.186:0/416296612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2e40b1e30 0x7fa2e41ec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:13.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 -- 172.21.15.186:0/416296612 >> 172.21.15.186:0/416296612 conn(0x7fa2e40c8a20 msgr2=0x7fa2e40c6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:13.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 -- 172.21.15.186:0/416296612 shutdown_connections 2024-03-28T15:02:13.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:13.434+0000 7fa2ec62a640 1 -- 172.21.15.186:0/416296612 wait complete. 2024-03-28T15:02:13.447 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 9, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T15:01:31.425299+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4302": { "addr": "172.21.15.191:6834/185536353", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 185536353, "type": "v2" }, { "addr": "172.21.15.191:6834", "nonce": 185536353, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 21, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T15:02:11.169599+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/1278716016", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6833", "nonce": 1278716016, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1278716016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6835/2068761001", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2068761001, "type": "v2" }, { "addr": "172.21.15.186:6835", "nonce": 2068761001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2344737463", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6837", "nonce": 2344737463, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2344737463, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4284, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/1548903381", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 1548903381, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 1548903381, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T15:02:15.449 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:15.561 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.560+0000 7f7fdcb56640 1 Processor -- start 2024-03-28T15:02:15.561 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.561+0000 7f7fdcb56640 1 -- start start 2024-03-28T15:02:15.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.561+0000 7f7fdcb56640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fd8151990 0x7f7fd80c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:15.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fdcb56640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7fd80b1e30 0x7f7fd80c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:15.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fdcb56640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7fd80c2d00 0x7f7fd80712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:15.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fdcb56640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f7fd80c1da0 con 0x7f7fd80b1e30 2024-03-28T15:02:15.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fdcb56640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f7fd806caa0 con 0x7f7fd80c2d00 2024-03-28T15:02:15.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fdcb56640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7fd806cd80 con 0x7f7fd8151990 2024-03-28T15:02:15.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fd5d9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7fd80b1e30 0x7f7fd80c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:15.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fd5d9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7fd80b1e30 0x7f7fd80c2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43834/0 (socket says 172.21.15.186:43834) 2024-03-28T15:02:15.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fd5d9b640 1 -- 172.21.15.186:0/394095897 learned_addr learned my addr 172.21.15.186:0/394095897 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:15.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fd659c640 1 --2- 172.21.15.186:0/394095897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fd8151990 0x7f7fd80c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:15.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fd6d9d640 1 --2- 172.21.15.186:0/394095897 >> v2:172.21.15.191:3300/0 conn(0x7f7fd80c2d00 0x7f7fd80712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:15.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fd5d9b640 1 -- 172.21.15.186:0/394095897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fd8151990 msgr2=0x7f7fd80c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:15.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fd5d9b640 1 --2- 172.21.15.186:0/394095897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fd8151990 0x7f7fd80c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:15.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fd5d9b640 1 -- 172.21.15.186:0/394095897 >> v2:172.21.15.191:3300/0 conn(0x7f7fd80c2d00 msgr2=0x7f7fd80712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:15.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fd5d9b640 1 --2- 172.21.15.186:0/394095897 >> v2:172.21.15.191:3300/0 conn(0x7f7fd80c2d00 0x7f7fd80712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:15.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.562+0000 7f7fd5d9b640 1 -- 172.21.15.186:0/394095897 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd80c1340 con 0x7f7fd80b1e30 2024-03-28T15:02:15.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.563+0000 7f7fd5d9b640 1 --2- 172.21.15.186:0/394095897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fd80b1e30 0x7f7fd80c2770 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80ccfc0 tx=0x7f7fcc02e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9f8ccf6b2e4816 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:15.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.563+0000 7f7fd559a640 1 -- 172.21.15.186:0/394095897 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc03b070 con 0x7f7fd80b1e30 2024-03-28T15:02:15.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.563+0000 7f7fd559a640 1 -- 172.21.15.186:0/394095897 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fcc036440 con 0x7f7fd80b1e30 2024-03-28T15:02:15.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.563+0000 7f7fd559a640 1 -- 172.21.15.186:0/394095897 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc036810 con 0x7f7fd80b1e30 2024-03-28T15:02:15.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.564+0000 7f7fdcb56640 1 -- 172.21.15.186:0/394095897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fd80b1e30 msgr2=0x7f7fd80c2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:15.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.564+0000 7f7fdcb56640 1 --2- 172.21.15.186:0/394095897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fd80b1e30 0x7f7fd80c2770 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80ccfc0 tx=0x7f7fcc02e170 comp rx=0 tx=0).stop 2024-03-28T15:02:15.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.564+0000 7f7fdcb56640 1 -- 172.21.15.186:0/394095897 shutdown_connections 2024-03-28T15:02:15.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.564+0000 7f7fdcb56640 1 --2- 172.21.15.186:0/394095897 >> v2:172.21.15.191:3300/0 conn(0x7f7fd80c2d00 0x7f7fd80712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:15.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.564+0000 7f7fdcb56640 1 --2- 172.21.15.186:0/394095897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fd80b1e30 0x7f7fd80c2770 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:15.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.564+0000 7f7fdcb56640 1 --2- 172.21.15.186:0/394095897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fd8151990 0x7f7fd80c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:15.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.564+0000 7f7fdcb56640 1 -- 172.21.15.186:0/394095897 >> 172.21.15.186:0/394095897 conn(0x7f7fd80c8a20 msgr2=0x7f7fd80c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:15.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.564+0000 7f7fdcb56640 1 -- 172.21.15.186:0/394095897 shutdown_connections 2024-03-28T15:02:15.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.564+0000 7f7fdcb56640 1 -- 172.21.15.186:0/394095897 wait complete. 2024-03-28T15:02:15.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.565+0000 7f7fdcb56640 1 Processor -- start 2024-03-28T15:02:15.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.565+0000 7f7fdcb56640 1 -- start start 2024-03-28T15:02:15.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fdcb56640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fd80b1e30 0x7f7fd814d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:15.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fdcb56640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fd80c2d00 0x7f7fd80c3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:15.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fdcb56640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fd8151990 0x7f7fd81f0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fdcb56640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd8073dd0 con 0x7f7fd8151990 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fdcb56640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd80c1900 con 0x7f7fd80b1e30 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fdcb56640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7fd8073f50 con 0x7f7fd80c2d00 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fd6d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fd8151990 0x7f7fd81f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fd6d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fd8151990 0x7f7fd81f0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43848/0 (socket says 172.21.15.186:43848) 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fd6d9d640 1 -- 172.21.15.186:0/3066558837 learned_addr learned my addr 172.21.15.186:0/3066558837 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fd5d9b640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fd80c2d00 0x7f7fd80c3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fd659c640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fd80b1e30 0x7f7fd814d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fd6d9d640 1 -- 172.21.15.186:0/3066558837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fd80c2d00 msgr2=0x7f7fd80c3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fd6d9d640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fd80c2d00 0x7f7fd80c3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fd6d9d640 1 -- 172.21.15.186:0/3066558837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fd80b1e30 msgr2=0x7f7fd814d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fd6d9d640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fd80b1e30 0x7f7fd814d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fd6d9d640 1 -- 172.21.15.186:0/3066558837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd80c3c40 con 0x7f7fd8151990 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.566+0000 7f7fd659c640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fd80b1e30 0x7f7fd814d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:02:15.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.567+0000 7f7fd6d9d640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fd8151990 0x7f7fd81f0990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000efc0 tx=0x7f7fd000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:15.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.567+0000 7f7fc6ffd640 1 -- 172.21.15.186:0/3066558837 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0018020 con 0x7f7fd8151990 2024-03-28T15:02:15.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.567+0000 7f7fc6ffd640 1 -- 172.21.15.186:0/3066558837 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fd0014480 con 0x7f7fd8151990 2024-03-28T15:02:15.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.567+0000 7f7fdcb56640 1 -- 172.21.15.186:0/3066558837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd80aea60 con 0x7f7fd8151990 2024-03-28T15:02:15.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.567+0000 7f7fc6ffd640 1 -- 172.21.15.186:0/3066558837 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0014870 con 0x7f7fd8151990 2024-03-28T15:02:15.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.567+0000 7f7fdcb56640 1 -- 172.21.15.186:0/3066558837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd80af020 con 0x7f7fd8151990 2024-03-28T15:02:15.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.568+0000 7f7fdcb56640 1 -- 172.21.15.186:0/3066558837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd8069500 con 0x7f7fd8151990 2024-03-28T15:02:15.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.568+0000 7f7fc6ffd640 1 -- 172.21.15.186:0/3066558837 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7fd0005d10 con 0x7f7fd8151990 2024-03-28T15:02:15.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.569+0000 7f7fc6ffd640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7fb806c530 0x7f7fb806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:15.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.569+0000 7f7fc6ffd640 1 -- 172.21.15.186:0/3066558837 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5911+0+0 (secure 0 0 0) 0x7f7fd008c4f0 con 0x7f7fd8151990 2024-03-28T15:02:15.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.569+0000 7f7fd659c640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7fb806c530 0x7f7fb806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:15.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.569+0000 7f7fd659c640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7fb806c530 0x7f7fb806e9f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009a70 tx=0x7f7fc8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:15.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.573+0000 7f7fc6ffd640 1 -- 172.21.15.186:0/3066558837 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7fd0022030 con 0x7f7fd8151990 2024-03-28T15:02:15.787 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.787+0000 7f7fdcb56640 1 -- 172.21.15.186:0/3066558837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7fd8071240 con 0x7f7fd8151990 2024-03-28T15:02:15.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.788+0000 7f7fc6ffd640 1 -- 172.21.15.186:0/3066558837 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6006 (secure 0 0 0) 0x7f7fd0010040 con 0x7f7fd8151990 2024-03-28T15:02:15.788 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:15.788 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:11.169599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:02:15.788 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 9 2024-03-28T15:02:15.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 -- 172.21.15.186:0/3066558837 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7fb806c530 msgr2=0x7f7fb806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:15.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7fb806c530 0x7f7fb806e9f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009a70 tx=0x7f7fc8008040 comp rx=0 tx=0).stop 2024-03-28T15:02:15.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 -- 172.21.15.186:0/3066558837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fd8151990 msgr2=0x7f7fd81f0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:15.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fd8151990 0x7f7fd81f0990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000efc0 tx=0x7f7fd000c490 comp rx=0 tx=0).stop 2024-03-28T15:02:15.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 -- 172.21.15.186:0/3066558837 shutdown_connections 2024-03-28T15:02:15.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7fb806c530 0x7f7fb806e9f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:15.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fd8151990 0x7f7fd81f0990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:15.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fd80c2d00 0x7f7fd80c3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:15.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 --2- 172.21.15.186:0/3066558837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fd80b1e30 0x7f7fd814d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:15.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 -- 172.21.15.186:0/3066558837 >> 172.21.15.186:0/3066558837 conn(0x7f7fd80c8a20 msgr2=0x7f7fd80c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:15.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 -- 172.21.15.186:0/3066558837 shutdown_connections 2024-03-28T15:02:15.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:15.790+0000 7f7fdcb56640 1 -- 172.21.15.186:0/3066558837 wait complete. 2024-03-28T15:02:15.801 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:02:17.802 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:17.915 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.915+0000 7ff52d55c640 1 Processor -- start 2024-03-28T15:02:17.915 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.915+0000 7ff52d55c640 1 -- start start 2024-03-28T15:02:17.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.915+0000 7ff52d55c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff528154bd0 0x7ff52815a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:17.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.915+0000 7ff52d55c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5280c57e0 0x7ff5280cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:17.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.915+0000 7ff52d55c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5280c6e00 con 0x7ff5280c53c0 2024-03-28T15:02:17.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.915+0000 7ff52d55c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff5280c5ca0 con 0x7ff528154bd0 2024-03-28T15:02:17.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.915+0000 7ff52d55c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff52815b080 con 0x7ff5280c57e0 2024-03-28T15:02:17.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.916+0000 7ff526ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff528154bd0 0x7ff52815a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:17.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.916+0000 7ff526ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff528154bd0 0x7ff52815a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43854/0 (socket says 172.21.15.186:43854) 2024-03-28T15:02:17.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.916+0000 7ff526ffd640 1 -- 172.21.15.186:0/1170939846 learned_addr learned my addr 172.21.15.186:0/1170939846 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:17.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.916+0000 7ff5277fe640 1 --2- 172.21.15.186:0/1170939846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5280c57e0 0x7ff5280cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:17.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.916+0000 7ff526ffd640 1 -- 172.21.15.186:0/1170939846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5280c57e0 msgr2=0x7ff5280cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:17.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.916+0000 7ff526ffd640 1 --2- 172.21.15.186:0/1170939846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5280c57e0 0x7ff5280cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:17.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.916+0000 7ff526ffd640 1 -- 172.21.15.186:0/1170939846 >> v1:172.21.15.191:6789/0 conn(0x7ff5280c53c0 legacy=0x7ff5280c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:17.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.916+0000 7ff526ffd640 1 -- 172.21.15.186:0/1170939846 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52815b360 con 0x7ff528154bd0 2024-03-28T15:02:17.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.917+0000 7ff526ffd640 1 --2- 172.21.15.186:0/1170939846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff528154bd0 0x7ff52815a7d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff520009870 tx=0x7ff52002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2deb25e09a85d6d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:17.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.917+0000 7ff525ffb640 1 -- 172.21.15.186:0/1170939846 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52002f840 con 0x7ff528154bd0 2024-03-28T15:02:17.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.917+0000 7ff525ffb640 1 -- 172.21.15.186:0/1170939846 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff52002f9e0 con 0x7ff528154bd0 2024-03-28T15:02:17.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.917+0000 7ff525ffb640 1 -- 172.21.15.186:0/1170939846 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52002fdb0 con 0x7ff528154bd0 2024-03-28T15:02:17.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.917+0000 7ff52d55c640 1 -- 172.21.15.186:0/1170939846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff528154bd0 msgr2=0x7ff52815a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:17.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.917+0000 7ff52d55c640 1 --2- 172.21.15.186:0/1170939846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff528154bd0 0x7ff52815a7d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff520009870 tx=0x7ff52002ba20 comp rx=0 tx=0).stop 2024-03-28T15:02:17.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.917+0000 7ff52d55c640 1 -- 172.21.15.186:0/1170939846 shutdown_connections 2024-03-28T15:02:17.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.917+0000 7ff52d55c640 1 --2- 172.21.15.186:0/1170939846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5280c57e0 0x7ff5280cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:17.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.917+0000 7ff52d55c640 1 --2- 172.21.15.186:0/1170939846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff528154bd0 0x7ff52815a7d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:17.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.917+0000 7ff52d55c640 1 -- 172.21.15.186:0/1170939846 >> 172.21.15.186:0/1170939846 conn(0x7ff5280731f0 msgr2=0x7ff528073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:17.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.918+0000 7ff52d55c640 1 -- 172.21.15.186:0/1170939846 shutdown_connections 2024-03-28T15:02:17.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.918+0000 7ff52d55c640 1 -- 172.21.15.186:0/1170939846 wait complete. 2024-03-28T15:02:17.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.919+0000 7ff52d55c640 1 Processor -- start 2024-03-28T15:02:17.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.919+0000 7ff52d55c640 1 -- start start 2024-03-28T15:02:17.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.919+0000 7ff52d55c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5280c53c0 0x7ff5281f0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:17.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.919+0000 7ff52d55c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5280c57e0 0x7ff52814a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:17.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff52d55c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff528154bd0 0x7ff528145b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:17.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff52d55c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5280c61d0 con 0x7ff5280c53c0 2024-03-28T15:02:17.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff52d55c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff5280c5ef0 con 0x7ff5280c57e0 2024-03-28T15:02:17.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff52d55c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff5280cc390 con 0x7ff528154bd0 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff526ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5280c53c0 0x7ff5281f0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff526ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5280c53c0 0x7ff5281f0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43864/0 (socket says 172.21.15.186:43864) 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff526ffd640 1 -- 172.21.15.186:0/2210529646 learned_addr learned my addr 172.21.15.186:0/2210529646 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff5277fe640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff528154bd0 0x7ff528145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff5267fc640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5280c57e0 0x7ff52814a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff526ffd640 1 -- 172.21.15.186:0/2210529646 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff528154bd0 msgr2=0x7ff528145b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff526ffd640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff528154bd0 0x7ff528145b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff526ffd640 1 -- 172.21.15.186:0/2210529646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5280c57e0 msgr2=0x7ff52814a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff526ffd640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5280c57e0 0x7ff52814a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff526ffd640 1 -- 172.21.15.186:0/2210529646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5281f1370 con 0x7ff5280c53c0 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff5277fe640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff528154bd0 0x7ff528145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff526ffd640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5280c53c0 0x7ff5281f0e30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff52002ba60 tx=0x7ff520004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff50f7fe640 1 -- 172.21.15.186:0/2210529646 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff520047070 con 0x7ff5280c53c0 2024-03-28T15:02:17.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff50f7fe640 1 -- 172.21.15.186:0/2210529646 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff520037bc0 con 0x7ff5280c53c0 2024-03-28T15:02:17.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff52d55c640 1 -- 172.21.15.186:0/2210529646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff528146c60 con 0x7ff5280c53c0 2024-03-28T15:02:17.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff50f7fe640 1 -- 172.21.15.186:0/2210529646 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff520042480 con 0x7ff5280c53c0 2024-03-28T15:02:17.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.920+0000 7ff52d55c640 1 -- 172.21.15.186:0/2210529646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff528147220 con 0x7ff5280c53c0 2024-03-28T15:02:17.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.921+0000 7ff52d55c640 1 -- 172.21.15.186:0/2210529646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff528069500 con 0x7ff5280c53c0 2024-03-28T15:02:17.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.922+0000 7ff50f7fe640 1 -- 172.21.15.186:0/2210529646 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff52004a070 con 0x7ff5280c53c0 2024-03-28T15:02:17.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.922+0000 7ff50f7fe640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff51406c560 0x7ff51406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:17.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.922+0000 7ff5267fc640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff51406c560 0x7ff51406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:17.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.922+0000 7ff50f7fe640 1 -- 172.21.15.186:0/2210529646 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5911+0+0 (secure 0 0 0) 0x7ff5200ad7d0 con 0x7ff5280c53c0 2024-03-28T15:02:17.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.923+0000 7ff5267fc640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff51406c560 0x7ff51406ea20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff528149240 tx=0x7ff51c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:17.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:17.927+0000 7ff50f7fe640 1 -- 172.21.15.186:0/2210529646 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff520033030 con 0x7ff5280c53c0 2024-03-28T15:02:18.138 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.138+0000 7ff52d55c640 1 -- 172.21.15.186:0/2210529646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5280cbb60 con 0x7ff5280c53c0 2024-03-28T15:02:18.139 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.139+0000 7ff50f7fe640 1 -- 172.21.15.186:0/2210529646 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6012 (secure 0 0 0) 0x7ff52007bdb0 con 0x7ff5280c53c0 2024-03-28T15:02:18.139 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:18.139 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:16.179056+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:02:18.139 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 10 2024-03-28T15:02:18.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 -- 172.21.15.186:0/2210529646 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff51406c560 msgr2=0x7ff51406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:18.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff51406c560 0x7ff51406ea20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff528149240 tx=0x7ff51c009040 comp rx=0 tx=0).stop 2024-03-28T15:02:18.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 -- 172.21.15.186:0/2210529646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5280c53c0 msgr2=0x7ff5281f0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:18.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5280c53c0 0x7ff5281f0e30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff52002ba60 tx=0x7ff520004730 comp rx=0 tx=0).stop 2024-03-28T15:02:18.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 -- 172.21.15.186:0/2210529646 shutdown_connections 2024-03-28T15:02:18.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff51406c560 0x7ff51406ea20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:18.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff528154bd0 0x7ff528145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:18.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5280c57e0 0x7ff52814a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:18.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 --2- 172.21.15.186:0/2210529646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5280c53c0 0x7ff5281f0e30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:18.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 -- 172.21.15.186:0/2210529646 >> 172.21.15.186:0/2210529646 conn(0x7ff5280731f0 msgr2=0x7ff5280ce030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:18.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 -- 172.21.15.186:0/2210529646 shutdown_connections 2024-03-28T15:02:18.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:18.141+0000 7ff52d55c640 1 -- 172.21.15.186:0/2210529646 wait complete. 2024-03-28T15:02:18.152 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:02:20.153 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:20.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.264+0000 7f6dd7bf7640 1 Processor -- start 2024-03-28T15:02:20.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.265+0000 7f6dd7bf7640 1 -- start start 2024-03-28T15:02:20.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.265+0000 7f6dd7bf7640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6dd00b1e30 0x7f6dd00712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:20.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.265+0000 7f6dd7bf7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6dd00c2d00 0x7f6dd00717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:20.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.266+0000 7f6dd7bf7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dd006caa0 con 0x7f6dd0151990 2024-03-28T15:02:20.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.266+0000 7f6dd7bf7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6dd00c1da0 con 0x7f6dd00c2d00 2024-03-28T15:02:20.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.266+0000 7f6dd7bf7640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f6dd006cd80 con 0x7f6dd00b1e30 2024-03-28T15:02:20.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.266+0000 7f6dd6194640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6dd00c2d00 0x7f6dd00717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:20.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.266+0000 7f6dd6194640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6dd00c2d00 0x7f6dd00717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44892/0 (socket says 172.21.15.186:44892) 2024-03-28T15:02:20.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.266+0000 7f6dd6194640 1 -- 172.21.15.186:0/3395204354 learned_addr learned my addr 172.21.15.186:0/3395204354 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:20.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.266+0000 7f6dd5192640 1 --2- 172.21.15.186:0/3395204354 >> v2:172.21.15.191:3300/0 conn(0x7f6dd00b1e30 0x7f6dd00712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:20.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.266+0000 7f6dd6194640 1 -- 172.21.15.186:0/3395204354 >> v2:172.21.15.191:3300/0 conn(0x7f6dd00b1e30 msgr2=0x7f6dd00712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:20.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.266+0000 7f6dd6194640 1 --2- 172.21.15.186:0/3395204354 >> v2:172.21.15.191:3300/0 conn(0x7f6dd00b1e30 0x7f6dd00712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:20.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.266+0000 7f6dd6194640 1 -- 172.21.15.186:0/3395204354 >> v1:172.21.15.191:6789/0 conn(0x7f6dd0151990 legacy=0x7f6dd00c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:20.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.266+0000 7f6dd6194640 1 -- 172.21.15.186:0/3395204354 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd00c1340 con 0x7f6dd00c2d00 2024-03-28T15:02:20.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.267+0000 7f6dd6194640 1 --2- 172.21.15.186:0/3395204354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd00c2d00 0x7f6dd00717f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00724b0 tx=0x7f6dcc02e210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0720bf9dd1c2d0a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:20.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.267+0000 7f6dd4991640 1 -- 172.21.15.186:0/3395204354 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc00b0d0 con 0x7f6dd00c2d00 2024-03-28T15:02:20.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.267+0000 7f6dd4991640 1 -- 172.21.15.186:0/3395204354 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dcc00b270 con 0x7f6dd00c2d00 2024-03-28T15:02:20.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.267+0000 7f6dd4991640 1 -- 172.21.15.186:0/3395204354 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc009600 con 0x7f6dd00c2d00 2024-03-28T15:02:20.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.267+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/3395204354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd00c2d00 msgr2=0x7f6dd00717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:20.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.267+0000 7f6dd7bf7640 1 --2- 172.21.15.186:0/3395204354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd00c2d00 0x7f6dd00717f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00724b0 tx=0x7f6dcc02e210 comp rx=0 tx=0).stop 2024-03-28T15:02:20.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.267+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/3395204354 shutdown_connections 2024-03-28T15:02:20.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.267+0000 7f6dd7bf7640 1 --2- 172.21.15.186:0/3395204354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd00c2d00 0x7f6dd00717f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:20.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.267+0000 7f6dd7bf7640 1 --2- 172.21.15.186:0/3395204354 >> v2:172.21.15.191:3300/0 conn(0x7f6dd00b1e30 0x7f6dd00712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:20.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.267+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/3395204354 >> 172.21.15.186:0/3395204354 conn(0x7f6dd00c8a20 msgr2=0x7f6dd00c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:20.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.268+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/3395204354 shutdown_connections 2024-03-28T15:02:20.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.268+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/3395204354 wait complete. 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.269+0000 7f6dd7bf7640 1 Processor -- start 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.269+0000 7f6dd7bf7640 1 -- start start 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.269+0000 7f6dd7bf7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dd00b1e30 0x7f6dd01f10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.269+0000 7f6dd7bf7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dd00c2d00 0x7f6dd01f4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.269+0000 7f6dd7bf7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd0151990 0x7f6dd00cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.269+0000 7f6dd7bf7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd0071d30 con 0x7f6dd0151990 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.269+0000 7f6dd7bf7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd00c1900 con 0x7f6dd00c2d00 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.269+0000 7f6dd7bf7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f6dd00c2940 con 0x7f6dd00b1e30 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd6194640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd0151990 0x7f6dd00cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd6194640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd0151990 0x7f6dd00cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44900/0 (socket says 172.21.15.186:44900) 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd6194640 1 -- 172.21.15.186:0/667736441 learned_addr learned my addr 172.21.15.186:0/667736441 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd5192640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dd00c2d00 0x7f6dd01f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd5993640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dd00b1e30 0x7f6dd01f10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:20.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd6194640 1 -- 172.21.15.186:0/667736441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dd00b1e30 msgr2=0x7f6dd01f10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:20.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd6194640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dd00b1e30 0x7f6dd01f10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:20.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd6194640 1 -- 172.21.15.186:0/667736441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dd00c2d00 msgr2=0x7f6dd01f4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:20.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd6194640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dd00c2d00 0x7f6dd01f4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:20.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd6194640 1 -- 172.21.15.186:0/667736441 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd014d9a0 con 0x7f6dd0151990 2024-03-28T15:02:20.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd6194640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd0151990 0x7f6dd00cb110 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00d240 tx=0x7f6dcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:20.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dc67fc640 1 -- 172.21.15.186:0/667736441 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc00b0d0 con 0x7f6dd0151990 2024-03-28T15:02:20.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dc67fc640 1 -- 172.21.15.186:0/667736441 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dcc045070 con 0x7f6dd0151990 2024-03-28T15:02:20.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/667736441 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd014dc40 con 0x7f6dd0151990 2024-03-28T15:02:20.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dc67fc640 1 -- 172.21.15.186:0/667736441 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc00b720 con 0x7f6dd0151990 2024-03-28T15:02:20.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.270+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/667736441 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dd01f08b0 con 0x7f6dd0151990 2024-03-28T15:02:20.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.271+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/667736441 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dd0069500 con 0x7f6dd0151990 2024-03-28T15:02:20.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.272+0000 7f6dc67fc640 1 -- 172.21.15.186:0/667736441 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6dcc036bf0 con 0x7f6dd0151990 2024-03-28T15:02:20.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.272+0000 7f6dc67fc640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6db406c530 0x7f6db406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:20.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.272+0000 7f6dc67fc640 1 -- 172.21.15.186:0/667736441 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5911+0+0 (secure 0 0 0) 0x7f6dcc0ae720 con 0x7f6dd0151990 2024-03-28T15:02:20.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.272+0000 7f6dd5993640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6db406c530 0x7f6db406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:20.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.273+0000 7f6dd5993640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6db406c530 0x7f6db406e9f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80016e0 tx=0x7f6dc8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:20.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.277+0000 7f6dc67fc640 1 -- 172.21.15.186:0/667736441 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6dcc07c6b0 con 0x7f6dd0151990 2024-03-28T15:02:20.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.488+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/667736441 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6dd00b1a00 con 0x7f6dd0151990 2024-03-28T15:02:20.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.488+0000 7f6dc67fc640 1 -- 172.21.15.186:0/667736441 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6012 (secure 0 0 0) 0x7f6dcc0809b0 con 0x7f6dd0151990 2024-03-28T15:02:20.489 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:20.489 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:16.179056+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:02:20.489 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 10 2024-03-28T15:02:20.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.490+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/667736441 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6db406c530 msgr2=0x7f6db406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:20.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.490+0000 7f6dd7bf7640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6db406c530 0x7f6db406e9f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80016e0 tx=0x7f6dc8009040 comp rx=0 tx=0).stop 2024-03-28T15:02:20.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.490+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/667736441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd0151990 msgr2=0x7f6dd00cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:20.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.490+0000 7f6dd7bf7640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd0151990 0x7f6dd00cb110 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00d240 tx=0x7f6dcc004a40 comp rx=0 tx=0).stop 2024-03-28T15:02:20.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.490+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/667736441 shutdown_connections 2024-03-28T15:02:20.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.491+0000 7f6dd7bf7640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6db406c530 0x7f6db406e9f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:20.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.491+0000 7f6dd7bf7640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd0151990 0x7f6dd00cb110 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:20.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.491+0000 7f6dd7bf7640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dd00c2d00 0x7f6dd01f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:20.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.491+0000 7f6dd7bf7640 1 --2- 172.21.15.186:0/667736441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dd00b1e30 0x7f6dd01f10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:20.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.491+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/667736441 >> 172.21.15.186:0/667736441 conn(0x7f6dd00c8a20 msgr2=0x7f6dd0150290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:20.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.491+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/667736441 shutdown_connections 2024-03-28T15:02:20.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:20.491+0000 7f6dd7bf7640 1 -- 172.21.15.186:0/667736441 wait complete. 2024-03-28T15:02:20.502 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:02:22.503 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:22.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.615+0000 7ff2d346f640 1 Processor -- start 2024-03-28T15:02:22.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.616+0000 7ff2d346f640 1 -- start start 2024-03-28T15:02:22.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.616+0000 7ff2d346f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff2cc0c53c0 0x7ff2cc0c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:22.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.616+0000 7ff2d346f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff2cc0c57e0 0x7ff2cc0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:22.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.616+0000 7ff2d346f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2cc15b080 con 0x7ff2cc154bd0 2024-03-28T15:02:22.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.616+0000 7ff2d346f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff2cc0c5ca0 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.616+0000 7ff2d346f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7ff2cc15b360 con 0x7ff2cc0c53c0 2024-03-28T15:02:22.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.617+0000 7ff2d1a0c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff2cc0c57e0 0x7ff2cc0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:22.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.617+0000 7ff2d1a0c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff2cc0c57e0 0x7ff2cc0cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44906/0 (socket says 172.21.15.186:44906) 2024-03-28T15:02:22.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.617+0000 7ff2d1a0c640 1 -- 172.21.15.186:0/266169940 learned_addr learned my addr 172.21.15.186:0/266169940 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:22.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.617+0000 7ff2d0a0a640 1 --2- 172.21.15.186:0/266169940 >> v2:172.21.15.191:3300/0 conn(0x7ff2cc0c53c0 0x7ff2cc0c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:22.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.617+0000 7ff2d1a0c640 1 -- 172.21.15.186:0/266169940 >> v2:172.21.15.191:3300/0 conn(0x7ff2cc0c53c0 msgr2=0x7ff2cc0c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:22.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.617+0000 7ff2d1a0c640 1 --2- 172.21.15.186:0/266169940 >> v2:172.21.15.191:3300/0 conn(0x7ff2cc0c53c0 0x7ff2cc0c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:22.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.617+0000 7ff2d1a0c640 1 -- 172.21.15.186:0/266169940 >> v1:172.21.15.191:6789/0 conn(0x7ff2cc154bd0 legacy=0x7ff2cc15a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:22.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.617+0000 7ff2d1a0c640 1 -- 172.21.15.186:0/266169940 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc15b640 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.618+0000 7ff2d1a0c640 1 --2- 172.21.15.186:0/266169940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2cc0c57e0 0x7ff2cc0cb040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc08d6b0 tx=0x7ff2bc02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18379fb776f666a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.618+0000 7ff2cbfff640 1 -- 172.21.15.186:0/266169940 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc02f850 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.618+0000 7ff2cbfff640 1 -- 172.21.15.186:0/266169940 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2bc02f9f0 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.618+0000 7ff2cbfff640 1 -- 172.21.15.186:0/266169940 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc02fdc0 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.618+0000 7ff2d346f640 1 -- 172.21.15.186:0/266169940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2cc0c57e0 msgr2=0x7ff2cc0cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.618+0000 7ff2d346f640 1 --2- 172.21.15.186:0/266169940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2cc0c57e0 0x7ff2cc0cb040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc08d6b0 tx=0x7ff2bc02ba30 comp rx=0 tx=0).stop 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.618+0000 7ff2d346f640 1 -- 172.21.15.186:0/266169940 shutdown_connections 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.618+0000 7ff2d346f640 1 --2- 172.21.15.186:0/266169940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2cc0c57e0 0x7ff2cc0cb040 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.618+0000 7ff2d346f640 1 --2- 172.21.15.186:0/266169940 >> v2:172.21.15.191:3300/0 conn(0x7ff2cc0c53c0 0x7ff2cc0c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.618+0000 7ff2d346f640 1 -- 172.21.15.186:0/266169940 >> 172.21.15.186:0/266169940 conn(0x7ff2cc0731f0 msgr2=0x7ff2cc073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.619+0000 7ff2d346f640 1 -- 172.21.15.186:0/266169940 shutdown_connections 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.619+0000 7ff2d346f640 1 -- 172.21.15.186:0/266169940 wait complete. 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.620+0000 7ff2d346f640 1 Processor -- start 2024-03-28T15:02:22.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.620+0000 7ff2d346f640 1 -- start start 2024-03-28T15:02:22.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.620+0000 7ff2d346f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2cc0c53c0 0x7ff2cc159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:22.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.620+0000 7ff2d346f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2cc0c57e0 0x7ff2cc0b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:22.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.620+0000 7ff2d346f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2cc154bd0 0x7ff2cc1f0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:22.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.620+0000 7ff2d346f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc0cc390 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.620+0000 7ff2d346f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc0c61d0 con 0x7ff2cc154bd0 2024-03-28T15:02:22.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.620+0000 7ff2d346f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff2cc0c6c30 con 0x7ff2cc0c53c0 2024-03-28T15:02:22.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d0a0a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2cc0c57e0 0x7ff2cc0b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:22.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d0a0a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2cc0c57e0 0x7ff2cc0b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44912/0 (socket says 172.21.15.186:44912) 2024-03-28T15:02:22.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d0a0a640 1 -- 172.21.15.186:0/1061514175 learned_addr learned my addr 172.21.15.186:0/1061514175 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:22.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d1a0c640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2cc154bd0 0x7ff2cc1f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d120b640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2cc0c53c0 0x7ff2cc159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d0a0a640 1 -- 172.21.15.186:0/1061514175 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2cc0c53c0 msgr2=0x7ff2cc159ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d0a0a640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2cc0c53c0 0x7ff2cc159ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d0a0a640 1 -- 172.21.15.186:0/1061514175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2cc154bd0 msgr2=0x7ff2cc1f0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d0a0a640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2cc154bd0 0x7ff2cc1f0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d0a0a640 1 -- 172.21.15.186:0/1061514175 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc159650 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d0a0a640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2cc0c57e0 0x7ff2cc0b5ae0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000e8a0 tx=0x7ff2c000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2c9ffb640 1 -- 172.21.15.186:0/1061514175 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0017070 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2c9ffb640 1 -- 172.21.15.186:0/1061514175 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2c000c940 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d346f640 1 -- 172.21.15.186:0/1061514175 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2cc15a0e0 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2c9ffb640 1 -- 172.21.15.186:0/1061514175 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c000cd30 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.621+0000 7ff2d346f640 1 -- 172.21.15.186:0/1061514175 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2cc15e3b0 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.622+0000 7ff2d346f640 1 -- 172.21.15.186:0/1061514175 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2cc15ea60 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.623+0000 7ff2c9ffb640 1 -- 172.21.15.186:0/1061514175 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff2c001a070 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.623+0000 7ff2c9ffb640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff2b806c530 0x7ff2b806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:22.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.623+0000 7ff2d120b640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff2b806c530 0x7ff2b806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:22.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.624+0000 7ff2c9ffb640 1 -- 172.21.15.186:0/1061514175 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5911+0+0 (secure 0 0 0) 0x7ff2c008b9a0 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.624+0000 7ff2d120b640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff2b806c530 0x7ff2b806e9f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4001f10 tx=0x7ff2c4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:22.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.628+0000 7ff2c9ffb640 1 -- 172.21.15.186:0/1061514175 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff2c0059810 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.841+0000 7ff2d346f640 1 -- 172.21.15.186:0/1061514175 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2cc0cbb60 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.841+0000 7ff2c9ffb640 1 -- 172.21.15.186:0/1061514175 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6009 (secure 0 0 0) 0x7ff2c005db10 con 0x7ff2cc0c57e0 2024-03-28T15:02:22.842 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:22.842 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:21.233634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:02:22.842 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 11 2024-03-28T15:02:22.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.843+0000 7ff2d346f640 1 -- 172.21.15.186:0/1061514175 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff2b806c530 msgr2=0x7ff2b806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:22.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.843+0000 7ff2d346f640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff2b806c530 0x7ff2b806e9f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4001f10 tx=0x7ff2c4009040 comp rx=0 tx=0).stop 2024-03-28T15:02:22.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.843+0000 7ff2d346f640 1 -- 172.21.15.186:0/1061514175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2cc0c57e0 msgr2=0x7ff2cc0b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:22.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.843+0000 7ff2d346f640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2cc0c57e0 0x7ff2cc0b5ae0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000e8a0 tx=0x7ff2c000ed60 comp rx=0 tx=0).stop 2024-03-28T15:02:22.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.843+0000 7ff2d346f640 1 -- 172.21.15.186:0/1061514175 shutdown_connections 2024-03-28T15:02:22.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.843+0000 7ff2d346f640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff2b806c530 0x7ff2b806e9f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:22.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.844+0000 7ff2d346f640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2cc154bd0 0x7ff2cc1f0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:22.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.844+0000 7ff2d346f640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2cc0c57e0 0x7ff2cc0b5ae0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:22.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.844+0000 7ff2d346f640 1 --2- 172.21.15.186:0/1061514175 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2cc0c53c0 0x7ff2cc159ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:22.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.844+0000 7ff2d346f640 1 -- 172.21.15.186:0/1061514175 >> 172.21.15.186:0/1061514175 conn(0x7ff2cc0731f0 msgr2=0x7ff2cc0d00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:22.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.844+0000 7ff2d346f640 1 -- 172.21.15.186:0/1061514175 shutdown_connections 2024-03-28T15:02:22.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:22.844+0000 7ff2d346f640 1 -- 172.21.15.186:0/1061514175 wait complete. 2024-03-28T15:02:22.855 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:02:24.856 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:24.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.968+0000 7faa2b2a4640 1 Processor -- start 2024-03-28T15:02:24.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.968+0000 7faa2b2a4640 1 -- start start 2024-03-28T15:02:24.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa2b2a4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa24152810 0x7faa24158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:24.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa2b2a4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faa24071340 0x7faa2406c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:24.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa2b2a4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa24072df0 con 0x7faa2406ce60 2024-03-28T15:02:24.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa2b2a4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7faa24159500 con 0x7faa24071340 2024-03-28T15:02:24.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa2b2a4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7faa240c8660 con 0x7faa24152810 2024-03-28T15:02:24.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa29040640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa24152810 0x7faa24158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:24.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa2883f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faa24071340 0x7faa2406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:24.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa29841640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7faa2406ce60 0x7faa240c6cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:38618/0 (socket says 172.21.15.186:38618) 2024-03-28T15:02:24.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa29841640 1 -- 172.21.15.186:0/1965358074 learned_addr learned my addr 172.21.15.186:0/1965358074 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:02:24.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa29040640 1 -- 172.21.15.186:0/1965358074 >> v2:172.21.15.191:3300/0 conn(0x7faa24071340 msgr2=0x7faa2406c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:24.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa29040640 1 --2- 172.21.15.186:0/1965358074 >> v2:172.21.15.191:3300/0 conn(0x7faa24071340 0x7faa2406c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:24.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa29040640 1 -- 172.21.15.186:0/1965358074 >> v1:172.21.15.191:6789/0 conn(0x7faa2406ce60 legacy=0x7faa240c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:24.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.969+0000 7faa29040640 1 -- 172.21.15.186:0/1965358074 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa24071d50 con 0x7faa24152810 2024-03-28T15:02:24.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.970+0000 7faa29040640 1 --2- 172.21.15.186:0/1965358074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa24152810 0x7faa24158fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faa200097a0 tx=0x7faa2002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9af02717ad6eff69 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:24.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.970+0000 7faa1bfff640 1 -- 172.21.15.186:0/1965358074 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20035020 con 0x7faa24152810 2024-03-28T15:02:24.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.970+0000 7faa1bfff640 1 -- 172.21.15.186:0/1965358074 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa20031830 con 0x7faa24152810 2024-03-28T15:02:24.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.970+0000 7faa1bfff640 1 -- 172.21.15.186:0/1965358074 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20031c00 con 0x7faa24152810 2024-03-28T15:02:24.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.971+0000 7faa2b2a4640 1 -- 172.21.15.186:0/1965358074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa24152810 msgr2=0x7faa24158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:24.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.971+0000 7faa2b2a4640 1 --2- 172.21.15.186:0/1965358074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa24152810 0x7faa24158fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faa200097a0 tx=0x7faa2002b8d0 comp rx=0 tx=0).stop 2024-03-28T15:02:24.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.971+0000 7faa2b2a4640 1 -- 172.21.15.186:0/1965358074 shutdown_connections 2024-03-28T15:02:24.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.971+0000 7faa2b2a4640 1 --2- 172.21.15.186:0/1965358074 >> v2:172.21.15.191:3300/0 conn(0x7faa24071340 0x7faa2406c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:24.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.971+0000 7faa2b2a4640 1 --2- 172.21.15.186:0/1965358074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa24152810 0x7faa24158fc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:24.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.971+0000 7faa2b2a4640 1 -- 172.21.15.186:0/1965358074 >> 172.21.15.186:0/1965358074 conn(0x7faa240c8a20 msgr2=0x7faa240c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:24.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.971+0000 7faa2b2a4640 1 -- 172.21.15.186:0/1965358074 shutdown_connections 2024-03-28T15:02:24.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.971+0000 7faa2b2a4640 1 -- 172.21.15.186:0/1965358074 wait complete. 2024-03-28T15:02:24.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.972+0000 7faa2b2a4640 1 Processor -- start 2024-03-28T15:02:24.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.972+0000 7faa2b2a4640 1 -- start start 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa2b2a4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2406ce60 0x7faa241f1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa2b2a4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa24071340 0x7faa24157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa2b2a4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa24152810 0x7faa240c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa2b2a4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faa240725f0 con 0x7faa2406ce60 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa2b2a4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faa24072310 con 0x7faa24071340 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa2b2a4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7faa240728d0 con 0x7faa24152810 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa29040640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2406ce60 0x7faa241f1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa29040640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2406ce60 0x7faa241f1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44920/0 (socket says 172.21.15.186:44920) 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa29040640 1 -- 172.21.15.186:0/332656912 learned_addr learned my addr 172.21.15.186:0/332656912 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa29841640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa24152810 0x7faa240c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa2883f640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa24071340 0x7faa24157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa29040640 1 -- 172.21.15.186:0/332656912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa24152810 msgr2=0x7faa240c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:24.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa29040640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa24152810 0x7faa240c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:24.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa29040640 1 -- 172.21.15.186:0/332656912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa24071340 msgr2=0x7faa24157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:24.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa29040640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa24071340 0x7faa24157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:24.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.973+0000 7faa29040640 1 -- 172.21.15.186:0/332656912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa241f1840 con 0x7faa2406ce60 2024-03-28T15:02:24.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.974+0000 7faa29040640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2406ce60 0x7faa241f1300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faa20009890 tx=0x7faa200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:24.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.974+0000 7faa19ffb640 1 -- 172.21.15.186:0/332656912 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20035040 con 0x7faa2406ce60 2024-03-28T15:02:24.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.974+0000 7faa19ffb640 1 -- 172.21.15.186:0/332656912 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa2003b920 con 0x7faa2406ce60 2024-03-28T15:02:24.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.974+0000 7faa2b2a4640 1 -- 172.21.15.186:0/332656912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa240c29e0 con 0x7faa2406ce60 2024-03-28T15:02:24.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.974+0000 7faa19ffb640 1 -- 172.21.15.186:0/332656912 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2003bd10 con 0x7faa2406ce60 2024-03-28T15:02:24.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.974+0000 7faa2b2a4640 1 -- 172.21.15.186:0/332656912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa240c12c0 con 0x7faa2406ce60 2024-03-28T15:02:24.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.975+0000 7faa2b2a4640 1 -- 172.21.15.186:0/332656912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa24069500 con 0x7faa2406ce60 2024-03-28T15:02:24.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.975+0000 7faa19ffb640 1 -- 172.21.15.186:0/332656912 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7faa20038020 con 0x7faa2406ce60 2024-03-28T15:02:24.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.976+0000 7faa19ffb640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa0806c560 0x7faa0806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:24.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.976+0000 7faa2883f640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa0806c560 0x7faa0806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:24.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.976+0000 7faa19ffb640 1 -- 172.21.15.186:0/332656912 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5911+0+0 (secure 0 0 0) 0x7faa200ad870 con 0x7faa2406ce60 2024-03-28T15:02:24.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.977+0000 7faa2883f640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa0806c560 0x7faa0806ea20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faa240c7640 tx=0x7faa14009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:24.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:24.981+0000 7faa19ffb640 1 -- 172.21.15.186:0/332656912 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7faa20031800 con 0x7faa2406ce60 2024-03-28T15:02:25.195 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.195+0000 7faa2b2a4640 1 -- 172.21.15.186:0/332656912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa240c5ab0 con 0x7faa2406ce60 2024-03-28T15:02:25.195 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.195+0000 7faa19ffb640 1 -- 172.21.15.186:0/332656912 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6009 (secure 0 0 0) 0x7faa2007b650 con 0x7faa2406ce60 2024-03-28T15:02:25.195 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:25.195 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:21.233634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:02:25.196 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 11 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 -- 172.21.15.186:0/332656912 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa0806c560 msgr2=0x7faa0806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa0806c560 0x7faa0806ea20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faa240c7640 tx=0x7faa14009190 comp rx=0 tx=0).stop 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 -- 172.21.15.186:0/332656912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2406ce60 msgr2=0x7faa241f1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2406ce60 0x7faa241f1300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faa20009890 tx=0x7faa200046c0 comp rx=0 tx=0).stop 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 -- 172.21.15.186:0/332656912 shutdown_connections 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faa0806c560 0x7faa0806ea20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa24152810 0x7faa240c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa24071340 0x7faa24157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 --2- 172.21.15.186:0/332656912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2406ce60 0x7faa241f1300 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 -- 172.21.15.186:0/332656912 >> 172.21.15.186:0/332656912 conn(0x7faa240c8a20 msgr2=0x7faa240c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 -- 172.21.15.186:0/332656912 shutdown_connections 2024-03-28T15:02:25.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:25.197+0000 7faa2b2a4640 1 -- 172.21.15.186:0/332656912 wait complete. 2024-03-28T15:02:25.211 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 11, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T15:01:31.425299+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 11, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4302": { "addr": "172.21.15.191:6834/185536353", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 185536353, "type": "v2" }, { "addr": "172.21.15.191:6834", "nonce": 185536353, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 21, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T15:02:21.233634+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/1278716016", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6833", "nonce": 1278716016, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1278716016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6835/2068761001", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2068761001, "type": "v2" }, { "addr": "172.21.15.186:6835", "nonce": 2068761001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2344737463", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6837", "nonce": 2344737463, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2344737463, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4284, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/1548903381", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 1548903381, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 1548903381, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T15:02:25.362 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:02:25.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:02:27.212 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:27.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.327+0000 7f83ba888640 1 Processor -- start 2024-03-28T15:02:27.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.327+0000 7f83ba888640 1 -- start start 2024-03-28T15:02:27.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.328+0000 7f83ba888640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f83b4154bd0 0x7f83b415a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:27.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.328+0000 7f83ba888640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b40c6e00 con 0x7f83b40c57e0 2024-03-28T15:02:27.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.328+0000 7f83ba888640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b415b080 con 0x7f83b40c53c0 2024-03-28T15:02:27.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.328+0000 7f83ba888640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f83b40c5ca0 con 0x7f83b4154bd0 2024-03-28T15:02:27.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.328+0000 7f83b8e25640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f83b40c57e0 0x7f83b40cb040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:36720/0 (socket says 172.21.15.186:36720) 2024-03-28T15:02:27.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.328+0000 7f83b8e25640 1 -- 172.21.15.186:0/3394048972 learned_addr learned my addr 172.21.15.186:0/3394048972 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:02:27.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b3fff640 1 --2- 172.21.15.186:0/3394048972 >> v2:172.21.15.191:3300/0 conn(0x7f83b4154bd0 0x7f83b415a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:27.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2994374875 0 0) 0x7f83b40c6e00 con 0x7f83b40c57e0 2024-03-28T15:02:27.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f839c003520 con 0x7f83b40c57e0 2024-03-28T15:02:27.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3707239031 0 0) 0x7f83b415b080 con 0x7f83b40c53c0 2024-03-28T15:02:27.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83b40c6e00 con 0x7f83b40c53c0 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 648524566 0 0) 0x7f839c003520 con 0x7f83b40c57e0 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 >> v2:172.21.15.191:3300/0 conn(0x7f83b4154bd0 msgr2=0x7f83b415a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 --2- 172.21.15.186:0/3394048972 >> v2:172.21.15.191:3300/0 conn(0x7f83b4154bd0 0x7f83b415a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 >> v1:172.21.15.191:6789/0 conn(0x7f83b40c53c0 legacy=0x7f83b40c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b415b360 con 0x7f83b40c57e0 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f83ac0019f0 con 0x7f83b40c57e0 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b3fff640 1 --2- 172.21.15.186:0/3394048972 >> v2:172.21.15.191:3300/0 conn(0x7f83b4154bd0 0x7f83b415a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 >> v1:172.21.15.186:6789/0 conn(0x7f83b40c57e0 legacy=0x7f83b40cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 --2- 172.21.15.186:0/3394048972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f839c003b40 0x7f839c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 --2- 172.21.15.186:0/3394048972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f839c004800 0x7f839c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 --2- 172.21.15.186:0/3394048972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f839c0051c0 0x7f839c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83b40c5ca0 con 0x7f839c0051c0 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f839c00bfb0 con 0x7f839c004800 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f839c00c1b0 con 0x7f839c003b40 2024-03-28T15:02:27.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b8e25640 1 --2- 172.21.15.186:0/3394048972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f839c0051c0 0x7f839c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b3fff640 1 --2- 172.21.15.186:0/3394048972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f839c003b40 0x7f839c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.329+0000 7f83b37fe640 1 --2- 172.21.15.186:0/3394048972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f839c004800 0x7f839c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.330+0000 7f83b8e25640 1 -- 172.21.15.186:0/3394048972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f839c003b40 msgr2=0x7f839c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.330+0000 7f83b8e25640 1 --2- 172.21.15.186:0/3394048972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f839c003b40 0x7f839c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.330+0000 7f83b8e25640 1 -- 172.21.15.186:0/3394048972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f839c004800 msgr2=0x7f839c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.330+0000 7f83b8e25640 1 --2- 172.21.15.186:0/3394048972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f839c004800 0x7f839c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.330+0000 7f83b8e25640 1 -- 172.21.15.186:0/3394048972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c003520 con 0x7f839c0051c0 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.330+0000 7f83b8e25640 1 -- 172.21.15.186:0/3394048972 reap_dead start 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.330+0000 7f83b8e25640 1 --2- 172.21.15.186:0/3394048972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f839c0051c0 0x7f839c00b9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00a7c0 tx=0x7f839c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f38e0f8a192e1d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.330+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac00a410 con 0x7f839c0051c0 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.330+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83ac034630 con 0x7f839c0051c0 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.330+0000 7f83b2ffd640 1 -- 172.21.15.186:0/3394048972 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac034a20 con 0x7f839c0051c0 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.331+0000 7f83ba888640 1 -- 172.21.15.186:0/3394048972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f839c0051c0 msgr2=0x7f839c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:27.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.331+0000 7f83ba888640 1 --2- 172.21.15.186:0/3394048972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f839c0051c0 0x7f839c00b9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00a7c0 tx=0x7f839c005830 comp rx=0 tx=0).stop 2024-03-28T15:02:27.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.331+0000 7f83ba888640 1 -- 172.21.15.186:0/3394048972 shutdown_connections 2024-03-28T15:02:27.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.331+0000 7f83ba888640 1 --2- 172.21.15.186:0/3394048972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f839c0051c0 0x7f839c00b9c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:27.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.331+0000 7f83ba888640 1 -- 172.21.15.186:0/3394048972 >> 172.21.15.186:0/3394048972 conn(0x7f83b40731f0 msgr2=0x7f83b4073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:27.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.331+0000 7f83ba888640 1 -- 172.21.15.186:0/3394048972 shutdown_connections 2024-03-28T15:02:27.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.331+0000 7f83ba888640 1 -- 172.21.15.186:0/3394048972 wait complete. 2024-03-28T15:02:27.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.332+0000 7f83ba888640 1 Processor -- start 2024-03-28T15:02:27.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.332+0000 7f83ba888640 1 -- start start 2024-03-28T15:02:27.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.333+0000 7f83ba888640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f839c0051c0 0x7f83b40b5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:27.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.333+0000 7f83ba888640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83b415b990 0x7f83b41f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:27.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.333+0000 7f83ba888640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b41f14f0 0x7f83b4159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:27.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.333+0000 7f83ba888640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83b40c61d0 con 0x7f83b41f14f0 2024-03-28T15:02:27.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.333+0000 7f83ba888640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f83b40c5ef0 con 0x7f839c0051c0 2024-03-28T15:02:27.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.333+0000 7f83ba888640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f83b40cc390 con 0x7f83b415b990 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.333+0000 7f83b8e25640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b41f14f0 0x7f83b4159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.333+0000 7f83b8e25640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b41f14f0 0x7f83b4159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44936/0 (socket says 172.21.15.186:44936) 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.333+0000 7f83b8e25640 1 -- 172.21.15.186:0/3221546266 learned_addr learned my addr 172.21.15.186:0/3221546266 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.333+0000 7f83b37fe640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83b415b990 0x7f83b41f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.333+0000 7f83b3fff640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f839c0051c0 0x7f83b40b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83b8e25640 1 -- 172.21.15.186:0/3221546266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83b415b990 msgr2=0x7f83b41f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83b8e25640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83b415b990 0x7f83b41f0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83b8e25640 1 -- 172.21.15.186:0/3221546266 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f839c0051c0 msgr2=0x7f83b40b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83b8e25640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f839c0051c0 0x7f83b40b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83b8e25640 1 -- 172.21.15.186:0/3221546266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b41f1970 con 0x7f83b41f14f0 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83b3fff640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f839c0051c0 0x7f83b40b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83b8e25640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b41f14f0 0x7f83b4159690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f83ac001330 tx=0x7f83b40d0c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83b0ff9640 1 -- 172.21.15.186:0/3221546266 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac044070 con 0x7f83b41f14f0 2024-03-28T15:02:27.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83b0ff9640 1 -- 172.21.15.186:0/3221546266 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83ac004b60 con 0x7f83b41f14f0 2024-03-28T15:02:27.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83ba888640 1 -- 172.21.15.186:0/3221546266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b415e9b0 con 0x7f83b41f14f0 2024-03-28T15:02:27.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83b0ff9640 1 -- 172.21.15.186:0/3221546266 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac039040 con 0x7f83b41f14f0 2024-03-28T15:02:27.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.334+0000 7f83ba888640 1 -- 172.21.15.186:0/3221546266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b41f47b0 con 0x7f83b41f14f0 2024-03-28T15:02:27.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.335+0000 7f83ba888640 1 -- 172.21.15.186:0/3221546266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b415b080 con 0x7f83b41f14f0 2024-03-28T15:02:27.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.335+0000 7f83b0ff9640 1 -- 172.21.15.186:0/3221546266 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f83ac047070 con 0x7f83b41f14f0 2024-03-28T15:02:27.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.336+0000 7f83b0ff9640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f83a006c560 0x7f83a006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:27.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.336+0000 7f83b0ff9640 1 -- 172.21.15.186:0/3221546266 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5911+0+0 (secure 0 0 0) 0x7f83ac0ac910 con 0x7f83b41f14f0 2024-03-28T15:02:27.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.336+0000 7f83b3fff640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f83a006c560 0x7f83a006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:27.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.336+0000 7f83b3fff640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f83a006c560 0x7f83a006ea20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009cd0 tx=0x7f83a4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:27.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.340+0000 7f83b0ff9640 1 -- 172.21.15.186:0/3221546266 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f83b415b080 con 0x7f83b41f14f0 2024-03-28T15:02:27.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.552+0000 7f83ba888640 1 -- 172.21.15.186:0/3221546266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83b4069500 con 0x7f83b41f14f0 2024-03-28T15:02:27.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.553+0000 7f83b0ff9640 1 -- 172.21.15.186:0/3221546266 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6016 (secure 0 0 0) 0x7f83b41f1970 con 0x7f83b41f14f0 2024-03-28T15:02:27.553 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:27.553 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:02:27.553 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 12 2024-03-28T15:02:27.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 -- 172.21.15.186:0/3221546266 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f83a006c560 msgr2=0x7f83a006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:27.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f83a006c560 0x7f83a006ea20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009cd0 tx=0x7f83a4008040 comp rx=0 tx=0).stop 2024-03-28T15:02:27.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 -- 172.21.15.186:0/3221546266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b41f14f0 msgr2=0x7f83b4159690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:27.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b41f14f0 0x7f83b4159690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f83ac001330 tx=0x7f83b40d0c50 comp rx=0 tx=0).stop 2024-03-28T15:02:27.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 -- 172.21.15.186:0/3221546266 shutdown_connections 2024-03-28T15:02:27.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f83a006c560 0x7f83a006ea20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:27.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83b41f14f0 0x7f83b4159690 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:27.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83b415b990 0x7f83b41f0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:27.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 --2- 172.21.15.186:0/3221546266 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f839c0051c0 0x7f83b40b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:27.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 -- 172.21.15.186:0/3221546266 >> 172.21.15.186:0/3221546266 conn(0x7f83b40731f0 msgr2=0x7f83b40cce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:27.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 -- 172.21.15.186:0/3221546266 shutdown_connections 2024-03-28T15:02:27.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:27.555+0000 7f83ba888640 1 -- 172.21.15.186:0/3221546266 wait complete. 2024-03-28T15:02:27.566 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4308 2024-03-28T15:02:27.567 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.f 2024-03-28T15:02:32.567 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-03-28T15:02:32.567 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-28T15:02:32.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-03-28T15:02:32.570 INFO:tasks.ceph.mds.f:Started 2024-03-28T15:02:32.570 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:32.645 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2024-03-28T15:02:32.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.676+0000 7f8ec49b5640 1 Processor -- start 2024-03-28T15:02:32.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.676+0000 7f8ec49b5640 1 -- start start 2024-03-28T15:02:32.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.676+0000 7f8ec49b5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8ec0152810 0x7f8ec0158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:32.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.676+0000 7f8ec49b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec0071340 0x7f8ec006c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:32.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.676+0000 7f8ec49b5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ec0072df0 con 0x7f8ec006ce60 2024-03-28T15:02:32.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.676+0000 7f8ec49b5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8ec0159500 con 0x7f8ec0152810 2024-03-28T15:02:32.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.676+0000 7f8ec49b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8ec00c8660 con 0x7f8ec0071340 2024-03-28T15:02:32.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.677+0000 7f8ebed9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8ec006ce60 0x7f8ec00c6cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:35992/0 (socket says 172.21.15.186:35992) 2024-03-28T15:02:32.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.677+0000 7f8ebed9d640 1 -- 172.21.15.186:0/1547747727 learned_addr learned my addr 172.21.15.186:0/1547747727 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:02:32.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.677+0000 7f8ebe59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8ec0152810 0x7f8ec0158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:32.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.677+0000 7f8eb7fff640 1 --2- 172.21.15.186:0/1547747727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec0071340 0x7f8ec006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:32.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.677+0000 7f8ebe59c640 1 -- 172.21.15.186:0/1547747727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec0071340 msgr2=0x7f8ec006c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:32.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.677+0000 7f8ebe59c640 1 --2- 172.21.15.186:0/1547747727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec0071340 0x7f8ec006c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:32.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.677+0000 7f8ebe59c640 1 -- 172.21.15.186:0/1547747727 >> v1:172.21.15.186:6789/0 conn(0x7f8ec006ce60 legacy=0x7f8ec00c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:32.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.677+0000 7f8ebe59c640 1 -- 172.21.15.186:0/1547747727 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec0071d50 con 0x7f8ec0152810 2024-03-28T15:02:32.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.677+0000 7f8eb7fff640 1 --2- 172.21.15.186:0/1547747727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec0071340 0x7f8ec006c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:02:32.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ebe59c640 1 --2- 172.21.15.186:0/1547747727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0152810 0x7f8ec0158fc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8eb00097a0 tx=0x7f8eb002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e75929157ff303b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:32.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ebdd9b640 1 -- 172.21.15.186:0/1547747727 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0035020 con 0x7f8ec0152810 2024-03-28T15:02:32.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ebdd9b640 1 -- 172.21.15.186:0/1547747727 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eb0031830 con 0x7f8ec0152810 2024-03-28T15:02:32.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ebdd9b640 1 -- 172.21.15.186:0/1547747727 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0031c00 con 0x7f8ec0152810 2024-03-28T15:02:32.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1547747727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0152810 msgr2=0x7f8ec0158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:32.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ec49b5640 1 --2- 172.21.15.186:0/1547747727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0152810 0x7f8ec0158fc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8eb00097a0 tx=0x7f8eb002b8d0 comp rx=0 tx=0).stop 2024-03-28T15:02:32.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1547747727 shutdown_connections 2024-03-28T15:02:32.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ec49b5640 1 --2- 172.21.15.186:0/1547747727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec0071340 0x7f8ec006c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:32.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ec49b5640 1 --2- 172.21.15.186:0/1547747727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0152810 0x7f8ec0158fc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:32.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1547747727 >> 172.21.15.186:0/1547747727 conn(0x7f8ec00c8a20 msgr2=0x7f8ec00c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:32.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1547747727 shutdown_connections 2024-03-28T15:02:32.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.678+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1547747727 wait complete. 2024-03-28T15:02:32.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.680+0000 7f8ec49b5640 1 Processor -- start 2024-03-28T15:02:32.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.680+0000 7f8ec49b5640 1 -- start start 2024-03-28T15:02:32.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.680+0000 7f8ec49b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006ce60 0x7f8ec014cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:32.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ec49b5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec0071340 0x7f8ec01ef0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:32.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ec49b5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0152810 0x7f8ec01ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:32.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ec49b5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec00725f0 con 0x7f8ec0152810 2024-03-28T15:02:32.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ec49b5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec0072310 con 0x7f8ec0071340 2024-03-28T15:02:32.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ec49b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8ec00728d0 con 0x7f8ec006ce60 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ebed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0152810 0x7f8ec01ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ebed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0152810 0x7f8ec01ef600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40554/0 (socket says 172.21.15.186:40554) 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ebed9d640 1 -- 172.21.15.186:0/1996835822 learned_addr learned my addr 172.21.15.186:0/1996835822 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ebe59c640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006ce60 0x7f8ec014cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8eb7fff640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec0071340 0x7f8ec01ef0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ebed9d640 1 -- 172.21.15.186:0/1996835822 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006ce60 msgr2=0x7f8ec014cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ebed9d640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006ce60 0x7f8ec014cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ebed9d640 1 -- 172.21.15.186:0/1996835822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec0071340 msgr2=0x7f8ec01ef0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ebed9d640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec0071340 0x7f8ec01ef0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ebed9d640 1 -- 172.21.15.186:0/1996835822 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec0158700 con 0x7f8ec0152810 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ebed9d640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0152810 0x7f8ec01ef600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8eb800caa0 tx=0x7f8eb800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8eb6ffd640 1 -- 172.21.15.186:0/1996835822 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8017070 con 0x7f8ec0152810 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8eb6ffd640 1 -- 172.21.15.186:0/1996835822 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eb800ac00 con 0x7f8ec0152810 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1996835822 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec01f2e40 con 0x7f8ec0152810 2024-03-28T15:02:32.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8eb6ffd640 1 -- 172.21.15.186:0/1996835822 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8012400 con 0x7f8ec0152810 2024-03-28T15:02:32.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.681+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1996835822 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec0157e30 con 0x7f8ec0152810 2024-03-28T15:02:32.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.683+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1996835822 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ec0069500 con 0x7f8ec0152810 2024-03-28T15:02:32.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.683+0000 7f8eb6ffd640 1 -- 172.21.15.186:0/1996835822 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8eb801a070 con 0x7f8ec0152810 2024-03-28T15:02:32.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.683+0000 7f8eb6ffd640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8ea006c530 0x7f8ea006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:32.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.684+0000 7f8ebe59c640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8ea006c530 0x7f8ea006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:32.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.684+0000 7f8eb6ffd640 1 -- 172.21.15.186:0/1996835822 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5911+0+0 (secure 0 0 0) 0x7f8eb808b3e0 con 0x7f8ec0152810 2024-03-28T15:02:32.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.684+0000 7f8ebe59c640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8ea006c530 0x7f8ea006e9f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0036000 tx=0x7f8eb0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:32.688 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.689+0000 7f8eb6ffd640 1 -- 172.21.15.186:0/1996835822 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8eb8059250 con 0x7f8ec0152810 2024-03-28T15:02:32.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.909+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1996835822 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ec00c5ab0 con 0x7f8ec0152810 2024-03-28T15:02:32.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.910+0000 7f8eb6ffd640 1 -- 172.21.15.186:0/1996835822 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6016 (secure 0 0 0) 0x7f8eb805d550 con 0x7f8ec0152810 2024-03-28T15:02:32.910 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:32.910 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:02:32.910 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 12 2024-03-28T15:02:32.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1996835822 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8ea006c530 msgr2=0x7f8ea006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:32.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8ea006c530 0x7f8ea006e9f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0036000 tx=0x7f8eb0008030 comp rx=0 tx=0).stop 2024-03-28T15:02:32.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1996835822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0152810 msgr2=0x7f8ec01ef600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:32.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0152810 0x7f8ec01ef600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8eb800caa0 tx=0x7f8eb800cf60 comp rx=0 tx=0).stop 2024-03-28T15:02:32.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1996835822 shutdown_connections 2024-03-28T15:02:32.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8ea006c530 0x7f8ea006e9f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:32.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0152810 0x7f8ec01ef600 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:32.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec0071340 0x7f8ec01ef0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:32.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 --2- 172.21.15.186:0/1996835822 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006ce60 0x7f8ec014cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:32.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1996835822 >> 172.21.15.186:0/1996835822 conn(0x7f8ec00c8a20 msgr2=0x7f8ec00ca520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:32.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1996835822 shutdown_connections 2024-03-28T15:02:32.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:32.912+0000 7f8ec49b5640 1 -- 172.21.15.186:0/1996835822 wait complete. 2024-03-28T15:02:32.923 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2024-03-28T15:02:34.924 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:02:35.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.030+0000 7f9daa3c9640 1 Processor -- start 2024-03-28T15:02:35.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.030+0000 7f9daa3c9640 1 -- start start 2024-03-28T15:02:35.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9daa3c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9da40c5380 0x7f9da40cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:35.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9daa3c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9da40cb390 0x7f9da4165220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:35.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9daa3c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da40c6f90 con 0x7f9da40caa70 2024-03-28T15:02:35.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9daa3c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f9da4071a80 con 0x7f9da40c5380 2024-03-28T15:02:35.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9daa3c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f9da40c6030 con 0x7f9da40cb390 2024-03-28T15:02:35.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9da37fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9da40c5380 0x7f9da40cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9da3fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f9da40caa70 0x7f9da40c64c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:35996/0 (socket says 172.21.15.186:35996) 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9da37fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9da40c5380 0x7f9da40cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40558/0 (socket says 172.21.15.186:40558) 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9da3fff640 1 -- 172.21.15.186:0/4276787328 learned_addr learned my addr 172.21.15.186:0/4276787328 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9da8966640 1 --2- 172.21.15.186:0/4276787328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9da40cb390 0x7f9da4165220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9da37fe640 1 -- 172.21.15.186:0/4276787328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9da40cb390 msgr2=0x7f9da4165220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9da37fe640 1 --2- 172.21.15.186:0/4276787328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9da40cb390 0x7f9da4165220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9da37fe640 1 -- 172.21.15.186:0/4276787328 >> v1:172.21.15.186:6789/0 conn(0x7f9da40caa70 legacy=0x7f9da40c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9da37fe640 1 -- 172.21.15.186:0/4276787328 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da40cc970 con 0x7f9da40c5380 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.031+0000 7f9da2ffd640 1 -- 172.21.15.186:0/4276787328 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 31123967 0 0) 0x7f9d94001110 con 0x7f9da40caa70 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.032+0000 7f9da37fe640 1 --2- 172.21.15.186:0/4276787328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9da40c5380 0x7f9da40cae50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9d980097a0 tx=0x7f9d9802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e101d100f93501 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.032+0000 7f9da2ffd640 1 -- 172.21.15.186:0/4276787328 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d98035020 con 0x7f9da40c5380 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.032+0000 7f9da2ffd640 1 -- 172.21.15.186:0/4276787328 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d98031830 con 0x7f9da40c5380 2024-03-28T15:02:35.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.032+0000 7f9da2ffd640 1 -- 172.21.15.186:0/4276787328 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d98031c00 con 0x7f9da40c5380 2024-03-28T15:02:35.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.033+0000 7f9daa3c9640 1 -- 172.21.15.186:0/4276787328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9da40c5380 msgr2=0x7f9da40cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:35.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.033+0000 7f9daa3c9640 1 --2- 172.21.15.186:0/4276787328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9da40c5380 0x7f9da40cae50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9d980097a0 tx=0x7f9d9802b8d0 comp rx=0 tx=0).stop 2024-03-28T15:02:35.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.033+0000 7f9daa3c9640 1 -- 172.21.15.186:0/4276787328 shutdown_connections 2024-03-28T15:02:35.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.033+0000 7f9daa3c9640 1 --2- 172.21.15.186:0/4276787328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9da40cb390 0x7f9da4165220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:35.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.033+0000 7f9daa3c9640 1 --2- 172.21.15.186:0/4276787328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9da40c5380 0x7f9da40cae50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:35.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.033+0000 7f9daa3c9640 1 -- 172.21.15.186:0/4276787328 >> 172.21.15.186:0/4276787328 conn(0x7f9da40d00f0 msgr2=0x7f9da40d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:35.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.033+0000 7f9daa3c9640 1 -- 172.21.15.186:0/4276787328 shutdown_connections 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.033+0000 7f9daa3c9640 1 -- 172.21.15.186:0/4276787328 wait complete. 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.034+0000 7f9daa3c9640 1 Processor -- start 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.034+0000 7f9daa3c9640 1 -- start start 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9daa3c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9da40c5380 0x7f9da40cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9daa3c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9da40caa70 0x7f9da41f03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9daa3c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9da40cb390 0x7f9da414aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9daa3c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9da40c5cc0 con 0x7f9da40c5380 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9daa3c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9da415b530 con 0x7f9da40caa70 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9daa3c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f9da415b810 con 0x7f9da40cb390 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9da40c5380 0x7f9da40cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9da40c5380 0x7f9da40cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40570/0 (socket says 172.21.15.186:40570) 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da3fff640 1 -- 172.21.15.186:0/3958650088 learned_addr learned my addr 172.21.15.186:0/3958650088 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da8966640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9da40cb390 0x7f9da414aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da37fe640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9da40caa70 0x7f9da41f03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:35.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da3fff640 1 -- 172.21.15.186:0/3958650088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9da40cb390 msgr2=0x7f9da414aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:35.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da3fff640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9da40cb390 0x7f9da414aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:35.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da3fff640 1 -- 172.21.15.186:0/3958650088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9da40caa70 msgr2=0x7f9da41f03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:35.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da3fff640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9da40caa70 0x7f9da41f03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:35.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da3fff640 1 -- 172.21.15.186:0/3958650088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da4157b60 con 0x7f9da40c5380 2024-03-28T15:02:35.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da8966640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9da40cb390 0x7f9da414aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:02:35.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da3fff640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9da40c5380 0x7f9da40cf3a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400c7b0 tx=0x7f9d9400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:35.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da0ff9640 1 -- 172.21.15.186:0/3958650088 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d94018070 con 0x7f9da40c5380 2024-03-28T15:02:35.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da0ff9640 1 -- 172.21.15.186:0/3958650088 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d940072d0 con 0x7f9da40c5380 2024-03-28T15:02:35.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9daa3c9640 1 -- 172.21.15.186:0/3958650088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9da4156a50 con 0x7f9da40c5380 2024-03-28T15:02:35.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9da0ff9640 1 -- 172.21.15.186:0/3958650088 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d940047e0 con 0x7f9da40c5380 2024-03-28T15:02:35.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.035+0000 7f9daa3c9640 1 -- 172.21.15.186:0/3958650088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9da40c7590 con 0x7f9da40c5380 2024-03-28T15:02:35.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.036+0000 7f9daa3c9640 1 -- 172.21.15.186:0/3958650088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9da40b6380 con 0x7f9da40c5380 2024-03-28T15:02:35.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.037+0000 7f9da0ff9640 1 -- 172.21.15.186:0/3958650088 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9d9401b070 con 0x7f9da40c5380 2024-03-28T15:02:35.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.037+0000 7f9da0ff9640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9d9006c530 0x7f9d9006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:02:35.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.037+0000 7f9da0ff9640 1 -- 172.21.15.186:0/3958650088 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5911+0+0 (secure 0 0 0) 0x7f9d9408aaf0 con 0x7f9da40c5380 2024-03-28T15:02:35.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.037+0000 7f9da37fe640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9d9006c530 0x7f9d9006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:02:35.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.038+0000 7f9da37fe640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9d9006c530 0x7f9d9006e9f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9d98036000 tx=0x7f9d9803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:02:35.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.042+0000 7f9da0ff9640 1 -- 172.21.15.186:0/3958650088 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f9da40b6380 con 0x7f9da40c5380 2024-03-28T15:02:35.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.262+0000 7f9daa3c9640 1 -- 172.21.15.186:0/3958650088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9da40686d0 con 0x7f9da40c5380 2024-03-28T15:02:35.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.262+0000 7f9da0ff9640 1 -- 172.21.15.186:0/3958650088 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6844 (secure 0 0 0) 0x7f9da40c6f90 con 0x7f9da40c5380 2024-03-28T15:02:35.263 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:02:35.263 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:02:35.263 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 13 2024-03-28T15:02:35.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 -- 172.21.15.186:0/3958650088 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9d9006c530 msgr2=0x7f9d9006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:35.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9d9006c530 0x7f9d9006e9f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9d98036000 tx=0x7f9d9803d040 comp rx=0 tx=0).stop 2024-03-28T15:02:35.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 -- 172.21.15.186:0/3958650088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9da40c5380 msgr2=0x7f9da40cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:02:35.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9da40c5380 0x7f9da40cf3a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400c7b0 tx=0x7f9d9400cc70 comp rx=0 tx=0).stop 2024-03-28T15:02:35.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 -- 172.21.15.186:0/3958650088 shutdown_connections 2024-03-28T15:02:35.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9d9006c530 0x7f9d9006e9f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:35.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9da40cb390 0x7f9da414aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9da40caa70 0x7f9da41f03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 --2- 172.21.15.186:0/3958650088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9da40c5380 0x7f9da40cf3a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:02:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 -- 172.21.15.186:0/3958650088 >> 172.21.15.186:0/3958650088 conn(0x7f9da40d00f0 msgr2=0x7f9da4153e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:02:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 -- 172.21.15.186:0/3958650088 shutdown_connections 2024-03-28T15:02:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:02:35.266+0000 7f9daa3c9640 1 -- 172.21.15.186:0/3958650088 wait complete. 2024-03-28T15:02:35.278 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-03-28T15:02:35.278 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2024-03-28T15:02:36.066 INFO:tasks.ceph.mds.b.smithi191.stderr:2024-03-28T15:02:36.066+0000 7f1170b81640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T15:02:55.425 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:02:55.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:03:10.746 INFO:tasks.workunit.client.0.smithi186.stderr:Updating files: 99% (12210/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr:Note: switching to 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'. 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr: 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr: 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr: 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr: git switch -c 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr: 2024-03-28T15:03:11.451 INFO:tasks.workunit.client.0.smithi186.stderr:Or undo this operation with: 2024-03-28T15:03:11.452 INFO:tasks.workunit.client.0.smithi186.stderr: 2024-03-28T15:03:11.452 INFO:tasks.workunit.client.0.smithi186.stderr: git switch - 2024-03-28T15:03:11.452 INFO:tasks.workunit.client.0.smithi186.stderr: 2024-03-28T15:03:11.452 INFO:tasks.workunit.client.0.smithi186.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T15:03:11.452 INFO:tasks.workunit.client.0.smithi186.stderr: 2024-03-28T15:03:11.452 INFO:tasks.workunit.client.0.smithi186.stderr:HEAD is now at e0e48ba6ee8 Merge PR #56536 into wip-batrick-testing-20240327.230800 2024-03-28T15:03:11.456 DEBUG:teuthology.orchestra.run.smithi186:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T15:03:11.479 INFO:tasks.workunit.client.0.smithi186.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T15:03:11.481 INFO:tasks.workunit.client.0.smithi186.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T15:03:11.481 INFO:tasks.workunit.client.0.smithi186.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T15:03:12.095 INFO:tasks.workunit.client.0.smithi186.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T15:03:12.178 INFO:tasks.workunit.client.0.smithi186.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T15:03:12.220 INFO:tasks.workunit.client.0.smithi186.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T15:03:12.222 INFO:tasks.workunit.client.0.smithi186.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T15:03:12.222 INFO:tasks.workunit.client.0.smithi186.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T15:03:12.262 INFO:tasks.workunit.client.0.smithi186.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T15:03:12.266 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:03:12.266 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T15:03:12.323 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2024-03-28T15:03:12.325 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2024-03-28T15:03:12.326 DEBUG:teuthology.orchestra.run.smithi186:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2024-03-28T15:03:12.387 INFO:tasks.workunit.client.0.smithi186.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2024-03-28T15:03:14.899 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir foo 2024-03-28T15:03:14.906 INFO:tasks.workunit.client.0.smithi186.stderr:+ cp linux-2.6.33.tar.bz2 foo 2024-03-28T15:03:15.398 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir foo/.snap/barsnap 2024-03-28T15:03:15.417 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm foo/linux-2.6.33.tar.bz2 2024-03-28T15:03:15.430 INFO:tasks.workunit.client.0.smithi186.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2024-03-28T15:03:15.522 INFO:tasks.workunit.client.0.smithi186.stdout:passed: files are identical 2024-03-28T15:03:15.523 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 'passed: files are identical' 2024-03-28T15:03:15.523 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir foo/.snap/barsnap 2024-03-28T15:03:15.554 INFO:tasks.workunit.client.0.smithi186.stdout:OK 2024-03-28T15:03:15.555 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo OK 2024-03-28T15:03:15.555 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T15:03:15.555 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:03:15.640 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2024-03-28T15:03:15.641 DEBUG:teuthology.orchestra.run.smithi186:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2024-03-28T15:03:15.664 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 1 2024-03-28T15:03:15.665 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 2 2024-03-28T15:03:15.667 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 3 2024-03-28T15:03:15.668 INFO:tasks.workunit.client.0.smithi186.stderr:+ '[' -e file4 ']' 2024-03-28T15:03:15.668 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir .snap/snap1 2024-03-28T15:03:15.677 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 4 2024-03-28T15:03:15.679 INFO:tasks.workunit.client.0.smithi186.stderr:++ ls 2024-03-28T15:03:15.683 INFO:tasks.workunit.client.0.smithi186.stderr:+ now='file1 2024-03-28T15:03:15.683 INFO:tasks.workunit.client.0.smithi186.stderr:file2 2024-03-28T15:03:15.684 INFO:tasks.workunit.client.0.smithi186.stderr:file3 2024-03-28T15:03:15.684 INFO:tasks.workunit.client.0.smithi186.stderr:file4' 2024-03-28T15:03:15.684 INFO:tasks.workunit.client.0.smithi186.stderr:++ ls .snap/snap1 2024-03-28T15:03:15.688 INFO:tasks.workunit.client.0.smithi186.stderr:+ then='file1 2024-03-28T15:03:15.688 INFO:tasks.workunit.client.0.smithi186.stderr:file2 2024-03-28T15:03:15.688 INFO:tasks.workunit.client.0.smithi186.stderr:file3' 2024-03-28T15:03:15.689 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir .snap/snap1 2024-03-28T15:03:15.694 INFO:tasks.workunit.client.0.smithi186.stderr:+ '[' 'file1 2024-03-28T15:03:15.694 INFO:tasks.workunit.client.0.smithi186.stderr:file2 2024-03-28T15:03:15.694 INFO:tasks.workunit.client.0.smithi186.stderr:file3 2024-03-28T15:03:15.695 INFO:tasks.workunit.client.0.smithi186.stderr:file4' = 'file1 2024-03-28T15:03:15.695 INFO:tasks.workunit.client.0.smithi186.stderr:file2 2024-03-28T15:03:15.695 INFO:tasks.workunit.client.0.smithi186.stderr:file3' ']' 2024-03-28T15:03:15.695 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 1 2024-03-28T15:03:15.698 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 2 2024-03-28T15:03:15.707 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 3 2024-03-28T15:03:15.713 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir .snap/snap1 2024-03-28T15:03:15.732 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 4 2024-03-28T15:03:15.737 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir .snap/snap1 2024-03-28T15:03:15.745 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm file1 file2 file3 file4 2024-03-28T15:03:15.756 INFO:tasks.workunit.client.0.smithi186.stdout:OK 2024-03-28T15:03:15.757 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo OK 2024-03-28T15:03:15.757 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T15:03:15.757 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:03:15.833 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2024-03-28T15:03:15.834 DEBUG:teuthology.orchestra.run.smithi186:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2024-03-28T15:03:15.898 INFO:tasks.workunit.client.0.smithi186.stdout:Create dir 100 to 199 ... 2024-03-28T15:03:15.901 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 100 2024-03-28T15:03:15.923 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 101 2024-03-28T15:03:15.945 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 102 2024-03-28T15:03:15.965 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 103 2024-03-28T15:03:15.986 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 104 2024-03-28T15:03:16.006 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 105 2024-03-28T15:03:16.025 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 106 2024-03-28T15:03:16.045 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 107 2024-03-28T15:03:16.065 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 108 2024-03-28T15:03:16.085 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 109 2024-03-28T15:03:16.105 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 110 2024-03-28T15:03:16.127 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 111 2024-03-28T15:03:16.148 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 112 2024-03-28T15:03:16.168 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 113 2024-03-28T15:03:16.188 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 114 2024-03-28T15:03:16.209 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 115 2024-03-28T15:03:16.232 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 116 2024-03-28T15:03:16.254 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 117 2024-03-28T15:03:16.278 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 118 2024-03-28T15:03:16.302 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 119 2024-03-28T15:03:16.326 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 120 2024-03-28T15:03:16.365 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 121 2024-03-28T15:03:16.394 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 122 2024-03-28T15:03:16.415 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 123 2024-03-28T15:03:16.435 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 124 2024-03-28T15:03:16.455 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 125 2024-03-28T15:03:16.478 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 126 2024-03-28T15:03:16.500 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 127 2024-03-28T15:03:16.520 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 128 2024-03-28T15:03:16.540 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 129 2024-03-28T15:03:16.560 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 130 2024-03-28T15:03:16.582 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 131 2024-03-28T15:03:16.604 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 132 2024-03-28T15:03:16.624 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 133 2024-03-28T15:03:16.645 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 134 2024-03-28T15:03:16.666 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 135 2024-03-28T15:03:16.688 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 136 2024-03-28T15:03:16.709 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 137 2024-03-28T15:03:16.735 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 138 2024-03-28T15:03:16.757 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 139 2024-03-28T15:03:16.778 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 140 2024-03-28T15:03:16.800 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 141 2024-03-28T15:03:16.821 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 142 2024-03-28T15:03:16.841 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 143 2024-03-28T15:03:16.862 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 144 2024-03-28T15:03:16.882 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 145 2024-03-28T15:03:16.904 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 146 2024-03-28T15:03:16.926 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 147 2024-03-28T15:03:16.948 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 148 2024-03-28T15:03:16.968 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 149 2024-03-28T15:03:16.988 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 150 2024-03-28T15:03:17.009 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 151 2024-03-28T15:03:17.029 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 152 2024-03-28T15:03:17.049 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 153 2024-03-28T15:03:17.072 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 154 2024-03-28T15:03:17.092 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 155 2024-03-28T15:03:17.114 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 156 2024-03-28T15:03:17.135 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 157 2024-03-28T15:03:17.155 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 158 2024-03-28T15:03:17.175 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 159 2024-03-28T15:03:17.198 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 160 2024-03-28T15:03:17.219 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 161 2024-03-28T15:03:17.240 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 162 2024-03-28T15:03:17.262 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 163 2024-03-28T15:03:17.284 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 164 2024-03-28T15:03:17.310 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 165 2024-03-28T15:03:17.336 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 166 2024-03-28T15:03:17.357 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 167 2024-03-28T15:03:17.377 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 168 2024-03-28T15:03:17.397 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 169 2024-03-28T15:03:17.417 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 170 2024-03-28T15:03:17.439 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 171 2024-03-28T15:03:17.461 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 172 2024-03-28T15:03:17.481 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 173 2024-03-28T15:03:17.501 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 174 2024-03-28T15:03:17.522 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 175 2024-03-28T15:03:17.542 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 176 2024-03-28T15:03:17.563 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 177 2024-03-28T15:03:17.584 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 178 2024-03-28T15:03:17.605 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 179 2024-03-28T15:03:17.627 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 180 2024-03-28T15:03:17.646 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 181 2024-03-28T15:03:17.667 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 182 2024-03-28T15:03:17.902 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 183 2024-03-28T15:03:17.921 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 184 2024-03-28T15:03:17.944 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 185 2024-03-28T15:03:17.963 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 186 2024-03-28T15:03:17.985 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 187 2024-03-28T15:03:18.005 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 188 2024-03-28T15:03:18.025 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 189 2024-03-28T15:03:18.046 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 190 2024-03-28T15:03:18.067 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 191 2024-03-28T15:03:18.088 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 192 2024-03-28T15:03:18.110 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 193 2024-03-28T15:03:18.135 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 194 2024-03-28T15:03:18.155 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 195 2024-03-28T15:03:18.175 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 196 2024-03-28T15:03:18.196 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 197 2024-03-28T15:03:18.218 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 198 2024-03-28T15:03:18.239 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 199 2024-03-28T15:03:18.261 INFO:tasks.workunit.client.0.smithi186.stdout:Take first snapshot .snap/test1 2024-03-28T15:03:18.415 INFO:tasks.workunit.client.0.smithi186.stdout:Create dir 200 to 299 ... 2024-03-28T15:03:18.416 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 200 2024-03-28T15:03:18.767 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 201 2024-03-28T15:03:18.788 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 202 2024-03-28T15:03:18.809 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 203 2024-03-28T15:03:18.828 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 204 2024-03-28T15:03:18.848 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 205 2024-03-28T15:03:18.867 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 206 2024-03-28T15:03:18.887 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 207 2024-03-28T15:03:18.908 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 208 2024-03-28T15:03:18.927 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 209 2024-03-28T15:03:18.947 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 210 2024-03-28T15:03:18.966 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 211 2024-03-28T15:03:18.986 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 212 2024-03-28T15:03:19.005 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 213 2024-03-28T15:03:19.026 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 214 2024-03-28T15:03:19.046 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 215 2024-03-28T15:03:19.064 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 216 2024-03-28T15:03:19.084 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 217 2024-03-28T15:03:19.103 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 218 2024-03-28T15:03:19.123 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 219 2024-03-28T15:03:19.144 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 220 2024-03-28T15:03:19.164 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 221 2024-03-28T15:03:19.183 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 222 2024-03-28T15:03:19.203 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 223 2024-03-28T15:03:19.221 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 224 2024-03-28T15:03:19.240 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 225 2024-03-28T15:03:19.259 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 226 2024-03-28T15:03:19.279 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 227 2024-03-28T15:03:19.298 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 228 2024-03-28T15:03:19.319 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 229 2024-03-28T15:03:19.339 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 230 2024-03-28T15:03:19.359 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 231 2024-03-28T15:03:19.378 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 232 2024-03-28T15:03:19.397 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 233 2024-03-28T15:03:19.416 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 234 2024-03-28T15:03:19.436 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 235 2024-03-28T15:03:19.455 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 236 2024-03-28T15:03:19.475 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 237 2024-03-28T15:03:19.494 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 238 2024-03-28T15:03:19.513 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 239 2024-03-28T15:03:19.533 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 240 2024-03-28T15:03:19.553 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 241 2024-03-28T15:03:19.573 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 242 2024-03-28T15:03:19.593 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 243 2024-03-28T15:03:19.611 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 244 2024-03-28T15:03:19.630 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 245 2024-03-28T15:03:19.650 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 246 2024-03-28T15:03:19.668 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 247 2024-03-28T15:03:19.687 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 248 2024-03-28T15:03:19.706 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 249 2024-03-28T15:03:19.725 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 250 2024-03-28T15:03:19.751 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 251 2024-03-28T15:03:21.312 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 252 2024-03-28T15:03:21.332 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 253 2024-03-28T15:03:21.354 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 254 2024-03-28T15:03:21.375 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 255 2024-03-28T15:03:21.395 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 256 2024-03-28T15:03:21.413 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 257 2024-03-28T15:03:21.432 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 258 2024-03-28T15:03:21.453 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 259 2024-03-28T15:03:21.474 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 260 2024-03-28T15:03:21.493 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 261 2024-03-28T15:03:21.512 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 262 2024-03-28T15:03:21.533 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 263 2024-03-28T15:03:21.552 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 264 2024-03-28T15:03:21.571 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 265 2024-03-28T15:03:21.591 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 266 2024-03-28T15:03:21.610 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 267 2024-03-28T15:03:21.631 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 268 2024-03-28T15:03:21.650 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 269 2024-03-28T15:03:21.671 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 270 2024-03-28T15:03:21.690 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 271 2024-03-28T15:03:21.709 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 272 2024-03-28T15:03:21.728 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 273 2024-03-28T15:03:21.748 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 274 2024-03-28T15:03:21.914 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 275 2024-03-28T15:03:21.932 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 276 2024-03-28T15:03:21.951 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 277 2024-03-28T15:03:21.970 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 278 2024-03-28T15:03:21.990 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 279 2024-03-28T15:03:22.008 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 280 2024-03-28T15:03:22.028 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 281 2024-03-28T15:03:22.048 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 282 2024-03-28T15:03:22.072 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 283 2024-03-28T15:03:22.093 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 284 2024-03-28T15:03:22.113 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 285 2024-03-28T15:03:22.136 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 286 2024-03-28T15:03:22.159 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 287 2024-03-28T15:03:22.182 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 288 2024-03-28T15:03:22.202 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 289 2024-03-28T15:03:22.224 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 290 2024-03-28T15:03:22.245 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 291 2024-03-28T15:03:22.267 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 292 2024-03-28T15:03:22.289 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 293 2024-03-28T15:03:22.311 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 294 2024-03-28T15:03:22.332 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 295 2024-03-28T15:03:22.354 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 296 2024-03-28T15:03:22.373 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 297 2024-03-28T15:03:22.393 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 298 2024-03-28T15:03:22.413 INFO:tasks.workunit.client.0.smithi186.stdout: create dir 299 2024-03-28T15:03:22.433 INFO:tasks.workunit.client.0.smithi186.stdout:Create a snapshot in every first level dir ... 2024-03-28T15:03:22.510 INFO:tasks.workunit.client.0.smithi186.stdout: create 100/.snap/snap-subdir-test 2024-03-28T15:03:22.528 INFO:tasks.workunit.client.0.smithi186.stdout: create 100/30 file after the snapshot 2024-03-28T15:03:22.530 INFO:tasks.workunit.client.0.smithi186.stdout: create 100/31 file after the snapshot 2024-03-28T15:03:22.531 INFO:tasks.workunit.client.0.smithi186.stdout: create 100/32 file after the snapshot 2024-03-28T15:03:22.533 INFO:tasks.workunit.client.0.smithi186.stdout: create 100/33 file after the snapshot 2024-03-28T15:03:22.535 INFO:tasks.workunit.client.0.smithi186.stdout: create 100/34 file after the snapshot 2024-03-28T15:03:22.536 INFO:tasks.workunit.client.0.smithi186.stdout: create 100/35 file after the snapshot 2024-03-28T15:03:22.538 INFO:tasks.workunit.client.0.smithi186.stdout: create 100/36 file after the snapshot 2024-03-28T15:03:22.540 INFO:tasks.workunit.client.0.smithi186.stdout: create 100/37 file after the snapshot 2024-03-28T15:03:22.541 INFO:tasks.workunit.client.0.smithi186.stdout: create 100/38 file after the snapshot 2024-03-28T15:03:22.543 INFO:tasks.workunit.client.0.smithi186.stdout: create 100/39 file after the snapshot 2024-03-28T15:03:22.544 INFO:tasks.workunit.client.0.smithi186.stdout: create 101/.snap/snap-subdir-test 2024-03-28T15:03:22.572 INFO:tasks.workunit.client.0.smithi186.stdout: create 101/30 file after the snapshot 2024-03-28T15:03:22.574 INFO:tasks.workunit.client.0.smithi186.stdout: create 101/31 file after the snapshot 2024-03-28T15:03:22.575 INFO:tasks.workunit.client.0.smithi186.stdout: create 101/32 file after the snapshot 2024-03-28T15:03:22.577 INFO:tasks.workunit.client.0.smithi186.stdout: create 101/33 file after the snapshot 2024-03-28T15:03:22.579 INFO:tasks.workunit.client.0.smithi186.stdout: create 101/34 file after the snapshot 2024-03-28T15:03:22.581 INFO:tasks.workunit.client.0.smithi186.stdout: create 101/35 file after the snapshot 2024-03-28T15:03:22.582 INFO:tasks.workunit.client.0.smithi186.stdout: create 101/36 file after the snapshot 2024-03-28T15:03:22.584 INFO:tasks.workunit.client.0.smithi186.stdout: create 101/37 file after the snapshot 2024-03-28T15:03:22.585 INFO:tasks.workunit.client.0.smithi186.stdout: create 101/38 file after the snapshot 2024-03-28T15:03:22.587 INFO:tasks.workunit.client.0.smithi186.stdout: create 101/39 file after the snapshot 2024-03-28T15:03:22.588 INFO:tasks.workunit.client.0.smithi186.stdout: create 102/.snap/snap-subdir-test 2024-03-28T15:03:22.608 INFO:tasks.workunit.client.0.smithi186.stdout: create 102/30 file after the snapshot 2024-03-28T15:03:22.610 INFO:tasks.workunit.client.0.smithi186.stdout: create 102/31 file after the snapshot 2024-03-28T15:03:22.611 INFO:tasks.workunit.client.0.smithi186.stdout: create 102/32 file after the snapshot 2024-03-28T15:03:22.613 INFO:tasks.workunit.client.0.smithi186.stdout: create 102/33 file after the snapshot 2024-03-28T15:03:22.615 INFO:tasks.workunit.client.0.smithi186.stdout: create 102/34 file after the snapshot 2024-03-28T15:03:22.616 INFO:tasks.workunit.client.0.smithi186.stdout: create 102/35 file after the snapshot 2024-03-28T15:03:22.618 INFO:tasks.workunit.client.0.smithi186.stdout: create 102/36 file after the snapshot 2024-03-28T15:03:22.620 INFO:tasks.workunit.client.0.smithi186.stdout: create 102/37 file after the snapshot 2024-03-28T15:03:22.621 INFO:tasks.workunit.client.0.smithi186.stdout: create 102/38 file after the snapshot 2024-03-28T15:03:22.623 INFO:tasks.workunit.client.0.smithi186.stdout: create 102/39 file after the snapshot 2024-03-28T15:03:22.625 INFO:tasks.workunit.client.0.smithi186.stdout: create 103/.snap/snap-subdir-test 2024-03-28T15:03:22.651 INFO:tasks.workunit.client.0.smithi186.stdout: create 103/30 file after the snapshot 2024-03-28T15:03:22.652 INFO:tasks.workunit.client.0.smithi186.stdout: create 103/31 file after the snapshot 2024-03-28T15:03:22.654 INFO:tasks.workunit.client.0.smithi186.stdout: create 103/32 file after the snapshot 2024-03-28T15:03:22.656 INFO:tasks.workunit.client.0.smithi186.stdout: create 103/33 file after the snapshot 2024-03-28T15:03:22.658 INFO:tasks.workunit.client.0.smithi186.stdout: create 103/34 file after the snapshot 2024-03-28T15:03:22.662 INFO:tasks.workunit.client.0.smithi186.stdout: create 103/35 file after the snapshot 2024-03-28T15:03:22.662 INFO:tasks.workunit.client.0.smithi186.stdout: create 103/36 file after the snapshot 2024-03-28T15:03:22.663 INFO:tasks.workunit.client.0.smithi186.stdout: create 103/37 file after the snapshot 2024-03-28T15:03:22.665 INFO:tasks.workunit.client.0.smithi186.stdout: create 103/38 file after the snapshot 2024-03-28T15:03:22.666 INFO:tasks.workunit.client.0.smithi186.stdout: create 103/39 file after the snapshot 2024-03-28T15:03:22.668 INFO:tasks.workunit.client.0.smithi186.stdout: create 104/.snap/snap-subdir-test 2024-03-28T15:03:22.698 INFO:tasks.workunit.client.0.smithi186.stdout: create 104/30 file after the snapshot 2024-03-28T15:03:22.700 INFO:tasks.workunit.client.0.smithi186.stdout: create 104/31 file after the snapshot 2024-03-28T15:03:22.702 INFO:tasks.workunit.client.0.smithi186.stdout: create 104/32 file after the snapshot 2024-03-28T15:03:22.703 INFO:tasks.workunit.client.0.smithi186.stdout: create 104/33 file after the snapshot 2024-03-28T15:03:22.705 INFO:tasks.workunit.client.0.smithi186.stdout: create 104/34 file after the snapshot 2024-03-28T15:03:22.707 INFO:tasks.workunit.client.0.smithi186.stdout: create 104/35 file after the snapshot 2024-03-28T15:03:22.708 INFO:tasks.workunit.client.0.smithi186.stdout: create 104/36 file after the snapshot 2024-03-28T15:03:22.710 INFO:tasks.workunit.client.0.smithi186.stdout: create 104/37 file after the snapshot 2024-03-28T15:03:22.712 INFO:tasks.workunit.client.0.smithi186.stdout: create 104/38 file after the snapshot 2024-03-28T15:03:22.714 INFO:tasks.workunit.client.0.smithi186.stdout: create 104/39 file after the snapshot 2024-03-28T15:03:22.716 INFO:tasks.workunit.client.0.smithi186.stdout: create 105/.snap/snap-subdir-test 2024-03-28T15:03:22.745 INFO:tasks.workunit.client.0.smithi186.stdout: create 105/30 file after the snapshot 2024-03-28T15:03:22.746 INFO:tasks.workunit.client.0.smithi186.stdout: create 105/31 file after the snapshot 2024-03-28T15:03:22.748 INFO:tasks.workunit.client.0.smithi186.stdout: create 105/32 file after the snapshot 2024-03-28T15:03:22.750 INFO:tasks.workunit.client.0.smithi186.stdout: create 105/33 file after the snapshot 2024-03-28T15:03:22.751 INFO:tasks.workunit.client.0.smithi186.stdout: create 105/34 file after the snapshot 2024-03-28T15:03:22.753 INFO:tasks.workunit.client.0.smithi186.stdout: create 105/35 file after the snapshot 2024-03-28T15:03:22.755 INFO:tasks.workunit.client.0.smithi186.stdout: create 105/36 file after the snapshot 2024-03-28T15:03:22.757 INFO:tasks.workunit.client.0.smithi186.stdout: create 105/37 file after the snapshot 2024-03-28T15:03:22.759 INFO:tasks.workunit.client.0.smithi186.stdout: create 105/38 file after the snapshot 2024-03-28T15:03:22.761 INFO:tasks.workunit.client.0.smithi186.stdout: create 105/39 file after the snapshot 2024-03-28T15:03:22.762 INFO:tasks.workunit.client.0.smithi186.stdout: create 106/.snap/snap-subdir-test 2024-03-28T15:03:23.074 INFO:tasks.workunit.client.0.smithi186.stdout: create 106/30 file after the snapshot 2024-03-28T15:03:23.075 INFO:tasks.workunit.client.0.smithi186.stdout: create 106/31 file after the snapshot 2024-03-28T15:03:23.077 INFO:tasks.workunit.client.0.smithi186.stdout: create 106/32 file after the snapshot 2024-03-28T15:03:23.079 INFO:tasks.workunit.client.0.smithi186.stdout: create 106/33 file after the snapshot 2024-03-28T15:03:23.082 INFO:tasks.workunit.client.0.smithi186.stdout: create 106/34 file after the snapshot 2024-03-28T15:03:23.083 INFO:tasks.workunit.client.0.smithi186.stdout: create 106/35 file after the snapshot 2024-03-28T15:03:23.085 INFO:tasks.workunit.client.0.smithi186.stdout: create 106/36 file after the snapshot 2024-03-28T15:03:23.087 INFO:tasks.workunit.client.0.smithi186.stdout: create 106/37 file after the snapshot 2024-03-28T15:03:23.089 INFO:tasks.workunit.client.0.smithi186.stdout: create 106/38 file after the snapshot 2024-03-28T15:03:23.091 INFO:tasks.workunit.client.0.smithi186.stdout: create 106/39 file after the snapshot 2024-03-28T15:03:23.093 INFO:tasks.workunit.client.0.smithi186.stdout: create 107/.snap/snap-subdir-test 2024-03-28T15:03:23.115 INFO:tasks.workunit.client.0.smithi186.stdout: create 107/30 file after the snapshot 2024-03-28T15:03:23.116 INFO:tasks.workunit.client.0.smithi186.stdout: create 107/31 file after the snapshot 2024-03-28T15:03:23.118 INFO:tasks.workunit.client.0.smithi186.stdout: create 107/32 file after the snapshot 2024-03-28T15:03:23.120 INFO:tasks.workunit.client.0.smithi186.stdout: create 107/33 file after the snapshot 2024-03-28T15:03:23.121 INFO:tasks.workunit.client.0.smithi186.stdout: create 107/34 file after the snapshot 2024-03-28T15:03:23.123 INFO:tasks.workunit.client.0.smithi186.stdout: create 107/35 file after the snapshot 2024-03-28T15:03:23.125 INFO:tasks.workunit.client.0.smithi186.stdout: create 107/36 file after the snapshot 2024-03-28T15:03:23.126 INFO:tasks.workunit.client.0.smithi186.stdout: create 107/37 file after the snapshot 2024-03-28T15:03:23.128 INFO:tasks.workunit.client.0.smithi186.stdout: create 107/38 file after the snapshot 2024-03-28T15:03:23.130 INFO:tasks.workunit.client.0.smithi186.stdout: create 107/39 file after the snapshot 2024-03-28T15:03:23.131 INFO:tasks.workunit.client.0.smithi186.stdout: create 108/.snap/snap-subdir-test 2024-03-28T15:03:23.150 INFO:tasks.workunit.client.0.smithi186.stdout: create 108/30 file after the snapshot 2024-03-28T15:03:23.152 INFO:tasks.workunit.client.0.smithi186.stdout: create 108/31 file after the snapshot 2024-03-28T15:03:23.153 INFO:tasks.workunit.client.0.smithi186.stdout: create 108/32 file after the snapshot 2024-03-28T15:03:23.155 INFO:tasks.workunit.client.0.smithi186.stdout: create 108/33 file after the snapshot 2024-03-28T15:03:23.157 INFO:tasks.workunit.client.0.smithi186.stdout: create 108/34 file after the snapshot 2024-03-28T15:03:23.158 INFO:tasks.workunit.client.0.smithi186.stdout: create 108/35 file after the snapshot 2024-03-28T15:03:23.160 INFO:tasks.workunit.client.0.smithi186.stdout: create 108/36 file after the snapshot 2024-03-28T15:03:23.162 INFO:tasks.workunit.client.0.smithi186.stdout: create 108/37 file after the snapshot 2024-03-28T15:03:23.163 INFO:tasks.workunit.client.0.smithi186.stdout: create 108/38 file after the snapshot 2024-03-28T15:03:23.165 INFO:tasks.workunit.client.0.smithi186.stdout: create 108/39 file after the snapshot 2024-03-28T15:03:23.167 INFO:tasks.workunit.client.0.smithi186.stdout: create 109/.snap/snap-subdir-test 2024-03-28T15:03:23.188 INFO:tasks.workunit.client.0.smithi186.stdout: create 109/30 file after the snapshot 2024-03-28T15:03:23.190 INFO:tasks.workunit.client.0.smithi186.stdout: create 109/31 file after the snapshot 2024-03-28T15:03:23.191 INFO:tasks.workunit.client.0.smithi186.stdout: create 109/32 file after the snapshot 2024-03-28T15:03:23.193 INFO:tasks.workunit.client.0.smithi186.stdout: create 109/33 file after the snapshot 2024-03-28T15:03:23.195 INFO:tasks.workunit.client.0.smithi186.stdout: create 109/34 file after the snapshot 2024-03-28T15:03:23.196 INFO:tasks.workunit.client.0.smithi186.stdout: create 109/35 file after the snapshot 2024-03-28T15:03:23.198 INFO:tasks.workunit.client.0.smithi186.stdout: create 109/36 file after the snapshot 2024-03-28T15:03:23.199 INFO:tasks.workunit.client.0.smithi186.stdout: create 109/37 file after the snapshot 2024-03-28T15:03:23.202 INFO:tasks.workunit.client.0.smithi186.stdout: create 109/38 file after the snapshot 2024-03-28T15:03:23.203 INFO:tasks.workunit.client.0.smithi186.stdout: create 109/39 file after the snapshot 2024-03-28T15:03:23.205 INFO:tasks.workunit.client.0.smithi186.stdout: create 110/.snap/snap-subdir-test 2024-03-28T15:03:23.235 INFO:tasks.workunit.client.0.smithi186.stdout: create 110/30 file after the snapshot 2024-03-28T15:03:23.237 INFO:tasks.workunit.client.0.smithi186.stdout: create 110/31 file after the snapshot 2024-03-28T15:03:23.239 INFO:tasks.workunit.client.0.smithi186.stdout: create 110/32 file after the snapshot 2024-03-28T15:03:23.241 INFO:tasks.workunit.client.0.smithi186.stdout: create 110/33 file after the snapshot 2024-03-28T15:03:23.243 INFO:tasks.workunit.client.0.smithi186.stdout: create 110/34 file after the snapshot 2024-03-28T15:03:23.244 INFO:tasks.workunit.client.0.smithi186.stdout: create 110/35 file after the snapshot 2024-03-28T15:03:23.246 INFO:tasks.workunit.client.0.smithi186.stdout: create 110/36 file after the snapshot 2024-03-28T15:03:23.248 INFO:tasks.workunit.client.0.smithi186.stdout: create 110/37 file after the snapshot 2024-03-28T15:03:23.250 INFO:tasks.workunit.client.0.smithi186.stdout: create 110/38 file after the snapshot 2024-03-28T15:03:23.251 INFO:tasks.workunit.client.0.smithi186.stdout: create 110/39 file after the snapshot 2024-03-28T15:03:23.253 INFO:tasks.workunit.client.0.smithi186.stdout: create 111/.snap/snap-subdir-test 2024-03-28T15:03:23.274 INFO:tasks.workunit.client.0.smithi186.stdout: create 111/30 file after the snapshot 2024-03-28T15:03:23.274 INFO:tasks.workunit.client.0.smithi186.stdout: create 111/31 file after the snapshot 2024-03-28T15:03:23.276 INFO:tasks.workunit.client.0.smithi186.stdout: create 111/32 file after the snapshot 2024-03-28T15:03:23.277 INFO:tasks.workunit.client.0.smithi186.stdout: create 111/33 file after the snapshot 2024-03-28T15:03:23.279 INFO:tasks.workunit.client.0.smithi186.stdout: create 111/34 file after the snapshot 2024-03-28T15:03:23.281 INFO:tasks.workunit.client.0.smithi186.stdout: create 111/35 file after the snapshot 2024-03-28T15:03:23.283 INFO:tasks.workunit.client.0.smithi186.stdout: create 111/36 file after the snapshot 2024-03-28T15:03:23.285 INFO:tasks.workunit.client.0.smithi186.stdout: create 111/37 file after the snapshot 2024-03-28T15:03:23.287 INFO:tasks.workunit.client.0.smithi186.stdout: create 111/38 file after the snapshot 2024-03-28T15:03:23.288 INFO:tasks.workunit.client.0.smithi186.stdout: create 111/39 file after the snapshot 2024-03-28T15:03:23.290 INFO:tasks.workunit.client.0.smithi186.stdout: create 112/.snap/snap-subdir-test 2024-03-28T15:03:23.319 INFO:tasks.workunit.client.0.smithi186.stdout: create 112/30 file after the snapshot 2024-03-28T15:03:23.321 INFO:tasks.workunit.client.0.smithi186.stdout: create 112/31 file after the snapshot 2024-03-28T15:03:23.323 INFO:tasks.workunit.client.0.smithi186.stdout: create 112/32 file after the snapshot 2024-03-28T15:03:23.325 INFO:tasks.workunit.client.0.smithi186.stdout: create 112/33 file after the snapshot 2024-03-28T15:03:23.327 INFO:tasks.workunit.client.0.smithi186.stdout: create 112/34 file after the snapshot 2024-03-28T15:03:23.329 INFO:tasks.workunit.client.0.smithi186.stdout: create 112/35 file after the snapshot 2024-03-28T15:03:23.331 INFO:tasks.workunit.client.0.smithi186.stdout: create 112/36 file after the snapshot 2024-03-28T15:03:23.333 INFO:tasks.workunit.client.0.smithi186.stdout: create 112/37 file after the snapshot 2024-03-28T15:03:23.335 INFO:tasks.workunit.client.0.smithi186.stdout: create 112/38 file after the snapshot 2024-03-28T15:03:23.337 INFO:tasks.workunit.client.0.smithi186.stdout: create 112/39 file after the snapshot 2024-03-28T15:03:23.339 INFO:tasks.workunit.client.0.smithi186.stdout: create 113/.snap/snap-subdir-test 2024-03-28T15:03:23.359 INFO:tasks.workunit.client.0.smithi186.stdout: create 113/30 file after the snapshot 2024-03-28T15:03:23.361 INFO:tasks.workunit.client.0.smithi186.stdout: create 113/31 file after the snapshot 2024-03-28T15:03:23.362 INFO:tasks.workunit.client.0.smithi186.stdout: create 113/32 file after the snapshot 2024-03-28T15:03:23.364 INFO:tasks.workunit.client.0.smithi186.stdout: create 113/33 file after the snapshot 2024-03-28T15:03:23.366 INFO:tasks.workunit.client.0.smithi186.stdout: create 113/34 file after the snapshot 2024-03-28T15:03:23.367 INFO:tasks.workunit.client.0.smithi186.stdout: create 113/35 file after the snapshot 2024-03-28T15:03:23.369 INFO:tasks.workunit.client.0.smithi186.stdout: create 113/36 file after the snapshot 2024-03-28T15:03:23.371 INFO:tasks.workunit.client.0.smithi186.stdout: create 113/37 file after the snapshot 2024-03-28T15:03:23.372 INFO:tasks.workunit.client.0.smithi186.stdout: create 113/38 file after the snapshot 2024-03-28T15:03:23.374 INFO:tasks.workunit.client.0.smithi186.stdout: create 113/39 file after the snapshot 2024-03-28T15:03:23.375 INFO:tasks.workunit.client.0.smithi186.stdout: create 114/.snap/snap-subdir-test 2024-03-28T15:03:23.403 INFO:tasks.workunit.client.0.smithi186.stdout: create 114/30 file after the snapshot 2024-03-28T15:03:23.404 INFO:tasks.workunit.client.0.smithi186.stdout: create 114/31 file after the snapshot 2024-03-28T15:03:23.406 INFO:tasks.workunit.client.0.smithi186.stdout: create 114/32 file after the snapshot 2024-03-28T15:03:23.408 INFO:tasks.workunit.client.0.smithi186.stdout: create 114/33 file after the snapshot 2024-03-28T15:03:23.410 INFO:tasks.workunit.client.0.smithi186.stdout: create 114/34 file after the snapshot 2024-03-28T15:03:23.412 INFO:tasks.workunit.client.0.smithi186.stdout: create 114/35 file after the snapshot 2024-03-28T15:03:23.413 INFO:tasks.workunit.client.0.smithi186.stdout: create 114/36 file after the snapshot 2024-03-28T15:03:23.415 INFO:tasks.workunit.client.0.smithi186.stdout: create 114/37 file after the snapshot 2024-03-28T15:03:23.417 INFO:tasks.workunit.client.0.smithi186.stdout: create 114/38 file after the snapshot 2024-03-28T15:03:23.419 INFO:tasks.workunit.client.0.smithi186.stdout: create 114/39 file after the snapshot 2024-03-28T15:03:23.421 INFO:tasks.workunit.client.0.smithi186.stdout: create 115/.snap/snap-subdir-test 2024-03-28T15:03:23.447 INFO:tasks.workunit.client.0.smithi186.stdout: create 115/30 file after the snapshot 2024-03-28T15:03:23.448 INFO:tasks.workunit.client.0.smithi186.stdout: create 115/31 file after the snapshot 2024-03-28T15:03:23.450 INFO:tasks.workunit.client.0.smithi186.stdout: create 115/32 file after the snapshot 2024-03-28T15:03:23.451 INFO:tasks.workunit.client.0.smithi186.stdout: create 115/33 file after the snapshot 2024-03-28T15:03:23.453 INFO:tasks.workunit.client.0.smithi186.stdout: create 115/34 file after the snapshot 2024-03-28T15:03:23.455 INFO:tasks.workunit.client.0.smithi186.stdout: create 115/35 file after the snapshot 2024-03-28T15:03:23.456 INFO:tasks.workunit.client.0.smithi186.stdout: create 115/36 file after the snapshot 2024-03-28T15:03:23.458 INFO:tasks.workunit.client.0.smithi186.stdout: create 115/37 file after the snapshot 2024-03-28T15:03:23.460 INFO:tasks.workunit.client.0.smithi186.stdout: create 115/38 file after the snapshot 2024-03-28T15:03:23.462 INFO:tasks.workunit.client.0.smithi186.stdout: create 115/39 file after the snapshot 2024-03-28T15:03:23.464 INFO:tasks.workunit.client.0.smithi186.stdout: create 116/.snap/snap-subdir-test 2024-03-28T15:03:23.483 INFO:tasks.workunit.client.0.smithi186.stdout: create 116/30 file after the snapshot 2024-03-28T15:03:23.485 INFO:tasks.workunit.client.0.smithi186.stdout: create 116/31 file after the snapshot 2024-03-28T15:03:23.487 INFO:tasks.workunit.client.0.smithi186.stdout: create 116/32 file after the snapshot 2024-03-28T15:03:23.489 INFO:tasks.workunit.client.0.smithi186.stdout: create 116/33 file after the snapshot 2024-03-28T15:03:23.490 INFO:tasks.workunit.client.0.smithi186.stdout: create 116/34 file after the snapshot 2024-03-28T15:03:23.492 INFO:tasks.workunit.client.0.smithi186.stdout: create 116/35 file after the snapshot 2024-03-28T15:03:23.494 INFO:tasks.workunit.client.0.smithi186.stdout: create 116/36 file after the snapshot 2024-03-28T15:03:23.496 INFO:tasks.workunit.client.0.smithi186.stdout: create 116/37 file after the snapshot 2024-03-28T15:03:23.498 INFO:tasks.workunit.client.0.smithi186.stdout: create 116/38 file after the snapshot 2024-03-28T15:03:23.500 INFO:tasks.workunit.client.0.smithi186.stdout: create 116/39 file after the snapshot 2024-03-28T15:03:23.501 INFO:tasks.workunit.client.0.smithi186.stdout: create 117/.snap/snap-subdir-test 2024-03-28T15:03:23.522 INFO:tasks.workunit.client.0.smithi186.stdout: create 117/30 file after the snapshot 2024-03-28T15:03:23.524 INFO:tasks.workunit.client.0.smithi186.stdout: create 117/31 file after the snapshot 2024-03-28T15:03:23.525 INFO:tasks.workunit.client.0.smithi186.stdout: create 117/32 file after the snapshot 2024-03-28T15:03:23.527 INFO:tasks.workunit.client.0.smithi186.stdout: create 117/33 file after the snapshot 2024-03-28T15:03:23.529 INFO:tasks.workunit.client.0.smithi186.stdout: create 117/34 file after the snapshot 2024-03-28T15:03:23.530 INFO:tasks.workunit.client.0.smithi186.stdout: create 117/35 file after the snapshot 2024-03-28T15:03:23.532 INFO:tasks.workunit.client.0.smithi186.stdout: create 117/36 file after the snapshot 2024-03-28T15:03:23.534 INFO:tasks.workunit.client.0.smithi186.stdout: create 117/37 file after the snapshot 2024-03-28T15:03:23.536 INFO:tasks.workunit.client.0.smithi186.stdout: create 117/38 file after the snapshot 2024-03-28T15:03:23.537 INFO:tasks.workunit.client.0.smithi186.stdout: create 117/39 file after the snapshot 2024-03-28T15:03:23.539 INFO:tasks.workunit.client.0.smithi186.stdout: create 118/.snap/snap-subdir-test 2024-03-28T15:03:23.558 INFO:tasks.workunit.client.0.smithi186.stdout: create 118/30 file after the snapshot 2024-03-28T15:03:23.560 INFO:tasks.workunit.client.0.smithi186.stdout: create 118/31 file after the snapshot 2024-03-28T15:03:23.562 INFO:tasks.workunit.client.0.smithi186.stdout: create 118/32 file after the snapshot 2024-03-28T15:03:23.564 INFO:tasks.workunit.client.0.smithi186.stdout: create 118/33 file after the snapshot 2024-03-28T15:03:23.566 INFO:tasks.workunit.client.0.smithi186.stdout: create 118/34 file after the snapshot 2024-03-28T15:03:23.568 INFO:tasks.workunit.client.0.smithi186.stdout: create 118/35 file after the snapshot 2024-03-28T15:03:23.569 INFO:tasks.workunit.client.0.smithi186.stdout: create 118/36 file after the snapshot 2024-03-28T15:03:23.571 INFO:tasks.workunit.client.0.smithi186.stdout: create 118/37 file after the snapshot 2024-03-28T15:03:23.573 INFO:tasks.workunit.client.0.smithi186.stdout: create 118/38 file after the snapshot 2024-03-28T15:03:23.575 INFO:tasks.workunit.client.0.smithi186.stdout: create 118/39 file after the snapshot 2024-03-28T15:03:23.577 INFO:tasks.workunit.client.0.smithi186.stdout: create 119/.snap/snap-subdir-test 2024-03-28T15:03:23.605 INFO:tasks.workunit.client.0.smithi186.stdout: create 119/30 file after the snapshot 2024-03-28T15:03:23.606 INFO:tasks.workunit.client.0.smithi186.stdout: create 119/31 file after the snapshot 2024-03-28T15:03:23.608 INFO:tasks.workunit.client.0.smithi186.stdout: create 119/32 file after the snapshot 2024-03-28T15:03:23.610 INFO:tasks.workunit.client.0.smithi186.stdout: create 119/33 file after the snapshot 2024-03-28T15:03:23.612 INFO:tasks.workunit.client.0.smithi186.stdout: create 119/34 file after the snapshot 2024-03-28T15:03:23.614 INFO:tasks.workunit.client.0.smithi186.stdout: create 119/35 file after the snapshot 2024-03-28T15:03:23.615 INFO:tasks.workunit.client.0.smithi186.stdout: create 119/36 file after the snapshot 2024-03-28T15:03:23.617 INFO:tasks.workunit.client.0.smithi186.stdout: create 119/37 file after the snapshot 2024-03-28T15:03:23.619 INFO:tasks.workunit.client.0.smithi186.stdout: create 119/38 file after the snapshot 2024-03-28T15:03:23.621 INFO:tasks.workunit.client.0.smithi186.stdout: create 119/39 file after the snapshot 2024-03-28T15:03:23.623 INFO:tasks.workunit.client.0.smithi186.stdout: create 120/.snap/snap-subdir-test 2024-03-28T15:03:23.643 INFO:tasks.workunit.client.0.smithi186.stdout: create 120/30 file after the snapshot 2024-03-28T15:03:23.645 INFO:tasks.workunit.client.0.smithi186.stdout: create 120/31 file after the snapshot 2024-03-28T15:03:23.647 INFO:tasks.workunit.client.0.smithi186.stdout: create 120/32 file after the snapshot 2024-03-28T15:03:23.648 INFO:tasks.workunit.client.0.smithi186.stdout: create 120/33 file after the snapshot 2024-03-28T15:03:23.650 INFO:tasks.workunit.client.0.smithi186.stdout: create 120/34 file after the snapshot 2024-03-28T15:03:23.652 INFO:tasks.workunit.client.0.smithi186.stdout: create 120/35 file after the snapshot 2024-03-28T15:03:23.654 INFO:tasks.workunit.client.0.smithi186.stdout: create 120/36 file after the snapshot 2024-03-28T15:03:23.655 INFO:tasks.workunit.client.0.smithi186.stdout: create 120/37 file after the snapshot 2024-03-28T15:03:23.657 INFO:tasks.workunit.client.0.smithi186.stdout: create 120/38 file after the snapshot 2024-03-28T15:03:23.659 INFO:tasks.workunit.client.0.smithi186.stdout: create 120/39 file after the snapshot 2024-03-28T15:03:23.661 INFO:tasks.workunit.client.0.smithi186.stdout: create 121/.snap/snap-subdir-test 2024-03-28T15:03:23.682 INFO:tasks.workunit.client.0.smithi186.stdout: create 121/30 file after the snapshot 2024-03-28T15:03:23.682 INFO:tasks.workunit.client.0.smithi186.stdout: create 121/31 file after the snapshot 2024-03-28T15:03:23.682 INFO:tasks.workunit.client.0.smithi186.stdout: create 121/32 file after the snapshot 2024-03-28T15:03:23.683 INFO:tasks.workunit.client.0.smithi186.stdout: create 121/33 file after the snapshot 2024-03-28T15:03:23.685 INFO:tasks.workunit.client.0.smithi186.stdout: create 121/34 file after the snapshot 2024-03-28T15:03:23.687 INFO:tasks.workunit.client.0.smithi186.stdout: create 121/35 file after the snapshot 2024-03-28T15:03:23.688 INFO:tasks.workunit.client.0.smithi186.stdout: create 121/36 file after the snapshot 2024-03-28T15:03:23.690 INFO:tasks.workunit.client.0.smithi186.stdout: create 121/37 file after the snapshot 2024-03-28T15:03:23.692 INFO:tasks.workunit.client.0.smithi186.stdout: create 121/38 file after the snapshot 2024-03-28T15:03:23.694 INFO:tasks.workunit.client.0.smithi186.stdout: create 121/39 file after the snapshot 2024-03-28T15:03:23.696 INFO:tasks.workunit.client.0.smithi186.stdout: create 122/.snap/snap-subdir-test 2024-03-28T15:03:23.716 INFO:tasks.workunit.client.0.smithi186.stdout: create 122/30 file after the snapshot 2024-03-28T15:03:23.717 INFO:tasks.workunit.client.0.smithi186.stdout: create 122/31 file after the snapshot 2024-03-28T15:03:23.719 INFO:tasks.workunit.client.0.smithi186.stdout: create 122/32 file after the snapshot 2024-03-28T15:03:23.721 INFO:tasks.workunit.client.0.smithi186.stdout: create 122/33 file after the snapshot 2024-03-28T15:03:23.723 INFO:tasks.workunit.client.0.smithi186.stdout: create 122/34 file after the snapshot 2024-03-28T15:03:23.725 INFO:tasks.workunit.client.0.smithi186.stdout: create 122/35 file after the snapshot 2024-03-28T15:03:23.727 INFO:tasks.workunit.client.0.smithi186.stdout: create 122/36 file after the snapshot 2024-03-28T15:03:23.729 INFO:tasks.workunit.client.0.smithi186.stdout: create 122/37 file after the snapshot 2024-03-28T15:03:23.730 INFO:tasks.workunit.client.0.smithi186.stdout: create 122/38 file after the snapshot 2024-03-28T15:03:23.732 INFO:tasks.workunit.client.0.smithi186.stdout: create 122/39 file after the snapshot 2024-03-28T15:03:23.734 INFO:tasks.workunit.client.0.smithi186.stdout: create 123/.snap/snap-subdir-test 2024-03-28T15:03:23.756 INFO:tasks.workunit.client.0.smithi186.stdout: create 123/30 file after the snapshot 2024-03-28T15:03:23.757 INFO:tasks.workunit.client.0.smithi186.stdout: create 123/31 file after the snapshot 2024-03-28T15:03:23.759 INFO:tasks.workunit.client.0.smithi186.stdout: create 123/32 file after the snapshot 2024-03-28T15:03:23.761 INFO:tasks.workunit.client.0.smithi186.stdout: create 123/33 file after the snapshot 2024-03-28T15:03:23.763 INFO:tasks.workunit.client.0.smithi186.stdout: create 123/34 file after the snapshot 2024-03-28T15:03:23.764 INFO:tasks.workunit.client.0.smithi186.stdout: create 123/35 file after the snapshot 2024-03-28T15:03:23.766 INFO:tasks.workunit.client.0.smithi186.stdout: create 123/36 file after the snapshot 2024-03-28T15:03:23.768 INFO:tasks.workunit.client.0.smithi186.stdout: create 123/37 file after the snapshot 2024-03-28T15:03:23.770 INFO:tasks.workunit.client.0.smithi186.stdout: create 123/38 file after the snapshot 2024-03-28T15:03:23.772 INFO:tasks.workunit.client.0.smithi186.stdout: create 123/39 file after the snapshot 2024-03-28T15:03:23.774 INFO:tasks.workunit.client.0.smithi186.stdout: create 124/.snap/snap-subdir-test 2024-03-28T15:03:23.796 INFO:tasks.workunit.client.0.smithi186.stdout: create 124/30 file after the snapshot 2024-03-28T15:03:23.859 INFO:tasks.workunit.client.0.smithi186.stdout: create 124/31 file after the snapshot 2024-03-28T15:03:23.861 INFO:tasks.workunit.client.0.smithi186.stdout: create 124/32 file after the snapshot 2024-03-28T15:03:23.863 INFO:tasks.workunit.client.0.smithi186.stdout: create 124/33 file after the snapshot 2024-03-28T15:03:23.864 INFO:tasks.workunit.client.0.smithi186.stdout: create 124/34 file after the snapshot 2024-03-28T15:03:23.866 INFO:tasks.workunit.client.0.smithi186.stdout: create 124/35 file after the snapshot 2024-03-28T15:03:23.868 INFO:tasks.workunit.client.0.smithi186.stdout: create 124/36 file after the snapshot 2024-03-28T15:03:23.869 INFO:tasks.workunit.client.0.smithi186.stdout: create 124/37 file after the snapshot 2024-03-28T15:03:23.871 INFO:tasks.workunit.client.0.smithi186.stdout: create 124/38 file after the snapshot 2024-03-28T15:03:23.873 INFO:tasks.workunit.client.0.smithi186.stdout: create 124/39 file after the snapshot 2024-03-28T15:03:23.875 INFO:tasks.workunit.client.0.smithi186.stdout: create 125/.snap/snap-subdir-test 2024-03-28T15:03:23.926 INFO:tasks.workunit.client.0.smithi186.stdout: create 125/30 file after the snapshot 2024-03-28T15:03:23.928 INFO:tasks.workunit.client.0.smithi186.stdout: create 125/31 file after the snapshot 2024-03-28T15:03:23.930 INFO:tasks.workunit.client.0.smithi186.stdout: create 125/32 file after the snapshot 2024-03-28T15:03:23.932 INFO:tasks.workunit.client.0.smithi186.stdout: create 125/33 file after the snapshot 2024-03-28T15:03:23.934 INFO:tasks.workunit.client.0.smithi186.stdout: create 125/34 file after the snapshot 2024-03-28T15:03:23.935 INFO:tasks.workunit.client.0.smithi186.stdout: create 125/35 file after the snapshot 2024-03-28T15:03:23.937 INFO:tasks.workunit.client.0.smithi186.stdout: create 125/36 file after the snapshot 2024-03-28T15:03:23.938 INFO:tasks.workunit.client.0.smithi186.stdout: create 125/37 file after the snapshot 2024-03-28T15:03:23.941 INFO:tasks.workunit.client.0.smithi186.stdout: create 125/38 file after the snapshot 2024-03-28T15:03:23.942 INFO:tasks.workunit.client.0.smithi186.stdout: create 125/39 file after the snapshot 2024-03-28T15:03:23.944 INFO:tasks.workunit.client.0.smithi186.stdout: create 126/.snap/snap-subdir-test 2024-03-28T15:03:23.963 INFO:tasks.workunit.client.0.smithi186.stdout: create 126/30 file after the snapshot 2024-03-28T15:03:23.964 INFO:tasks.workunit.client.0.smithi186.stdout: create 126/31 file after the snapshot 2024-03-28T15:03:23.966 INFO:tasks.workunit.client.0.smithi186.stdout: create 126/32 file after the snapshot 2024-03-28T15:03:23.967 INFO:tasks.workunit.client.0.smithi186.stdout: create 126/33 file after the snapshot 2024-03-28T15:03:23.969 INFO:tasks.workunit.client.0.smithi186.stdout: create 126/34 file after the snapshot 2024-03-28T15:03:23.971 INFO:tasks.workunit.client.0.smithi186.stdout: create 126/35 file after the snapshot 2024-03-28T15:03:23.973 INFO:tasks.workunit.client.0.smithi186.stdout: create 126/36 file after the snapshot 2024-03-28T15:03:23.974 INFO:tasks.workunit.client.0.smithi186.stdout: create 126/37 file after the snapshot 2024-03-28T15:03:23.976 INFO:tasks.workunit.client.0.smithi186.stdout: create 126/38 file after the snapshot 2024-03-28T15:03:23.979 INFO:tasks.workunit.client.0.smithi186.stdout: create 126/39 file after the snapshot 2024-03-28T15:03:23.981 INFO:tasks.workunit.client.0.smithi186.stdout: create 127/.snap/snap-subdir-test 2024-03-28T15:03:23.998 INFO:tasks.workunit.client.0.smithi186.stdout: create 127/30 file after the snapshot 2024-03-28T15:03:24.000 INFO:tasks.workunit.client.0.smithi186.stdout: create 127/31 file after the snapshot 2024-03-28T15:03:24.002 INFO:tasks.workunit.client.0.smithi186.stdout: create 127/32 file after the snapshot 2024-03-28T15:03:24.003 INFO:tasks.workunit.client.0.smithi186.stdout: create 127/33 file after the snapshot 2024-03-28T15:03:24.005 INFO:tasks.workunit.client.0.smithi186.stdout: create 127/34 file after the snapshot 2024-03-28T15:03:24.007 INFO:tasks.workunit.client.0.smithi186.stdout: create 127/35 file after the snapshot 2024-03-28T15:03:24.008 INFO:tasks.workunit.client.0.smithi186.stdout: create 127/36 file after the snapshot 2024-03-28T15:03:24.010 INFO:tasks.workunit.client.0.smithi186.stdout: create 127/37 file after the snapshot 2024-03-28T15:03:24.012 INFO:tasks.workunit.client.0.smithi186.stdout: create 127/38 file after the snapshot 2024-03-28T15:03:24.013 INFO:tasks.workunit.client.0.smithi186.stdout: create 127/39 file after the snapshot 2024-03-28T15:03:24.015 INFO:tasks.workunit.client.0.smithi186.stdout: create 128/.snap/snap-subdir-test 2024-03-28T15:03:24.034 INFO:tasks.workunit.client.0.smithi186.stdout: create 128/30 file after the snapshot 2024-03-28T15:03:24.036 INFO:tasks.workunit.client.0.smithi186.stdout: create 128/31 file after the snapshot 2024-03-28T15:03:24.037 INFO:tasks.workunit.client.0.smithi186.stdout: create 128/32 file after the snapshot 2024-03-28T15:03:24.039 INFO:tasks.workunit.client.0.smithi186.stdout: create 128/33 file after the snapshot 2024-03-28T15:03:24.041 INFO:tasks.workunit.client.0.smithi186.stdout: create 128/34 file after the snapshot 2024-03-28T15:03:24.043 INFO:tasks.workunit.client.0.smithi186.stdout: create 128/35 file after the snapshot 2024-03-28T15:03:24.045 INFO:tasks.workunit.client.0.smithi186.stdout: create 128/36 file after the snapshot 2024-03-28T15:03:24.047 INFO:tasks.workunit.client.0.smithi186.stdout: create 128/37 file after the snapshot 2024-03-28T15:03:24.049 INFO:tasks.workunit.client.0.smithi186.stdout: create 128/38 file after the snapshot 2024-03-28T15:03:24.051 INFO:tasks.workunit.client.0.smithi186.stdout: create 128/39 file after the snapshot 2024-03-28T15:03:24.052 INFO:tasks.workunit.client.0.smithi186.stdout: create 129/.snap/snap-subdir-test 2024-03-28T15:03:24.080 INFO:tasks.workunit.client.0.smithi186.stdout: create 129/30 file after the snapshot 2024-03-28T15:03:24.081 INFO:tasks.workunit.client.0.smithi186.stdout: create 129/31 file after the snapshot 2024-03-28T15:03:24.083 INFO:tasks.workunit.client.0.smithi186.stdout: create 129/32 file after the snapshot 2024-03-28T15:03:24.086 INFO:tasks.workunit.client.0.smithi186.stdout: create 129/33 file after the snapshot 2024-03-28T15:03:24.090 INFO:tasks.workunit.client.0.smithi186.stdout: create 129/34 file after the snapshot 2024-03-28T15:03:24.090 INFO:tasks.workunit.client.0.smithi186.stdout: create 129/35 file after the snapshot 2024-03-28T15:03:24.091 INFO:tasks.workunit.client.0.smithi186.stdout: create 129/36 file after the snapshot 2024-03-28T15:03:24.093 INFO:tasks.workunit.client.0.smithi186.stdout: create 129/37 file after the snapshot 2024-03-28T15:03:24.095 INFO:tasks.workunit.client.0.smithi186.stdout: create 129/38 file after the snapshot 2024-03-28T15:03:24.096 INFO:tasks.workunit.client.0.smithi186.stdout: create 129/39 file after the snapshot 2024-03-28T15:03:24.098 INFO:tasks.workunit.client.0.smithi186.stdout: create 130/.snap/snap-subdir-test 2024-03-28T15:03:24.119 INFO:tasks.workunit.client.0.smithi186.stdout: create 130/30 file after the snapshot 2024-03-28T15:03:24.121 INFO:tasks.workunit.client.0.smithi186.stdout: create 130/31 file after the snapshot 2024-03-28T15:03:24.122 INFO:tasks.workunit.client.0.smithi186.stdout: create 130/32 file after the snapshot 2024-03-28T15:03:24.124 INFO:tasks.workunit.client.0.smithi186.stdout: create 130/33 file after the snapshot 2024-03-28T15:03:24.126 INFO:tasks.workunit.client.0.smithi186.stdout: create 130/34 file after the snapshot 2024-03-28T15:03:24.127 INFO:tasks.workunit.client.0.smithi186.stdout: create 130/35 file after the snapshot 2024-03-28T15:03:24.129 INFO:tasks.workunit.client.0.smithi186.stdout: create 130/36 file after the snapshot 2024-03-28T15:03:24.130 INFO:tasks.workunit.client.0.smithi186.stdout: create 130/37 file after the snapshot 2024-03-28T15:03:24.132 INFO:tasks.workunit.client.0.smithi186.stdout: create 130/38 file after the snapshot 2024-03-28T15:03:24.134 INFO:tasks.workunit.client.0.smithi186.stdout: create 130/39 file after the snapshot 2024-03-28T15:03:24.136 INFO:tasks.workunit.client.0.smithi186.stdout: create 131/.snap/snap-subdir-test 2024-03-28T15:03:24.154 INFO:tasks.workunit.client.0.smithi186.stdout: create 131/30 file after the snapshot 2024-03-28T15:03:24.155 INFO:tasks.workunit.client.0.smithi186.stdout: create 131/31 file after the snapshot 2024-03-28T15:03:24.156 INFO:tasks.workunit.client.0.smithi186.stdout: create 131/32 file after the snapshot 2024-03-28T15:03:24.158 INFO:tasks.workunit.client.0.smithi186.stdout: create 131/33 file after the snapshot 2024-03-28T15:03:24.160 INFO:tasks.workunit.client.0.smithi186.stdout: create 131/34 file after the snapshot 2024-03-28T15:03:24.161 INFO:tasks.workunit.client.0.smithi186.stdout: create 131/35 file after the snapshot 2024-03-28T15:03:24.163 INFO:tasks.workunit.client.0.smithi186.stdout: create 131/36 file after the snapshot 2024-03-28T15:03:24.165 INFO:tasks.workunit.client.0.smithi186.stdout: create 131/37 file after the snapshot 2024-03-28T15:03:24.166 INFO:tasks.workunit.client.0.smithi186.stdout: create 131/38 file after the snapshot 2024-03-28T15:03:24.168 INFO:tasks.workunit.client.0.smithi186.stdout: create 131/39 file after the snapshot 2024-03-28T15:03:24.169 INFO:tasks.workunit.client.0.smithi186.stdout: create 132/.snap/snap-subdir-test 2024-03-28T15:03:24.197 INFO:tasks.workunit.client.0.smithi186.stdout: create 132/30 file after the snapshot 2024-03-28T15:03:24.198 INFO:tasks.workunit.client.0.smithi186.stdout: create 132/31 file after the snapshot 2024-03-28T15:03:24.200 INFO:tasks.workunit.client.0.smithi186.stdout: create 132/32 file after the snapshot 2024-03-28T15:03:24.202 INFO:tasks.workunit.client.0.smithi186.stdout: create 132/33 file after the snapshot 2024-03-28T15:03:24.204 INFO:tasks.workunit.client.0.smithi186.stdout: create 132/34 file after the snapshot 2024-03-28T15:03:24.205 INFO:tasks.workunit.client.0.smithi186.stdout: create 132/35 file after the snapshot 2024-03-28T15:03:24.207 INFO:tasks.workunit.client.0.smithi186.stdout: create 132/36 file after the snapshot 2024-03-28T15:03:24.209 INFO:tasks.workunit.client.0.smithi186.stdout: create 132/37 file after the snapshot 2024-03-28T15:03:24.211 INFO:tasks.workunit.client.0.smithi186.stdout: create 132/38 file after the snapshot 2024-03-28T15:03:24.212 INFO:tasks.workunit.client.0.smithi186.stdout: create 132/39 file after the snapshot 2024-03-28T15:03:24.214 INFO:tasks.workunit.client.0.smithi186.stdout: create 133/.snap/snap-subdir-test 2024-03-28T15:03:24.232 INFO:tasks.workunit.client.0.smithi186.stdout: create 133/30 file after the snapshot 2024-03-28T15:03:24.234 INFO:tasks.workunit.client.0.smithi186.stdout: create 133/31 file after the snapshot 2024-03-28T15:03:24.236 INFO:tasks.workunit.client.0.smithi186.stdout: create 133/32 file after the snapshot 2024-03-28T15:03:24.237 INFO:tasks.workunit.client.0.smithi186.stdout: create 133/33 file after the snapshot 2024-03-28T15:03:24.239 INFO:tasks.workunit.client.0.smithi186.stdout: create 133/34 file after the snapshot 2024-03-28T15:03:24.241 INFO:tasks.workunit.client.0.smithi186.stdout: create 133/35 file after the snapshot 2024-03-28T15:03:24.242 INFO:tasks.workunit.client.0.smithi186.stdout: create 133/36 file after the snapshot 2024-03-28T15:03:24.244 INFO:tasks.workunit.client.0.smithi186.stdout: create 133/37 file after the snapshot 2024-03-28T15:03:24.246 INFO:tasks.workunit.client.0.smithi186.stdout: create 133/38 file after the snapshot 2024-03-28T15:03:24.247 INFO:tasks.workunit.client.0.smithi186.stdout: create 133/39 file after the snapshot 2024-03-28T15:03:24.249 INFO:tasks.workunit.client.0.smithi186.stdout: create 134/.snap/snap-subdir-test 2024-03-28T15:03:24.267 INFO:tasks.workunit.client.0.smithi186.stdout: create 134/30 file after the snapshot 2024-03-28T15:03:24.269 INFO:tasks.workunit.client.0.smithi186.stdout: create 134/31 file after the snapshot 2024-03-28T15:03:24.271 INFO:tasks.workunit.client.0.smithi186.stdout: create 134/32 file after the snapshot 2024-03-28T15:03:24.273 INFO:tasks.workunit.client.0.smithi186.stdout: create 134/33 file after the snapshot 2024-03-28T15:03:24.274 INFO:tasks.workunit.client.0.smithi186.stdout: create 134/34 file after the snapshot 2024-03-28T15:03:24.276 INFO:tasks.workunit.client.0.smithi186.stdout: create 134/35 file after the snapshot 2024-03-28T15:03:24.278 INFO:tasks.workunit.client.0.smithi186.stdout: create 134/36 file after the snapshot 2024-03-28T15:03:24.279 INFO:tasks.workunit.client.0.smithi186.stdout: create 134/37 file after the snapshot 2024-03-28T15:03:24.281 INFO:tasks.workunit.client.0.smithi186.stdout: create 134/38 file after the snapshot 2024-03-28T15:03:24.284 INFO:tasks.workunit.client.0.smithi186.stdout: create 134/39 file after the snapshot 2024-03-28T15:03:24.286 INFO:tasks.workunit.client.0.smithi186.stdout: create 135/.snap/snap-subdir-test 2024-03-28T15:03:24.305 INFO:tasks.workunit.client.0.smithi186.stdout: create 135/30 file after the snapshot 2024-03-28T15:03:24.306 INFO:tasks.workunit.client.0.smithi186.stdout: create 135/31 file after the snapshot 2024-03-28T15:03:24.309 INFO:tasks.workunit.client.0.smithi186.stdout: create 135/32 file after the snapshot 2024-03-28T15:03:24.310 INFO:tasks.workunit.client.0.smithi186.stdout: create 135/33 file after the snapshot 2024-03-28T15:03:24.312 INFO:tasks.workunit.client.0.smithi186.stdout: create 135/34 file after the snapshot 2024-03-28T15:03:24.314 INFO:tasks.workunit.client.0.smithi186.stdout: create 135/35 file after the snapshot 2024-03-28T15:03:24.316 INFO:tasks.workunit.client.0.smithi186.stdout: create 135/36 file after the snapshot 2024-03-28T15:03:24.318 INFO:tasks.workunit.client.0.smithi186.stdout: create 135/37 file after the snapshot 2024-03-28T15:03:24.320 INFO:tasks.workunit.client.0.smithi186.stdout: create 135/38 file after the snapshot 2024-03-28T15:03:24.322 INFO:tasks.workunit.client.0.smithi186.stdout: create 135/39 file after the snapshot 2024-03-28T15:03:24.323 INFO:tasks.workunit.client.0.smithi186.stdout: create 136/.snap/snap-subdir-test 2024-03-28T15:03:24.350 INFO:tasks.workunit.client.0.smithi186.stdout: create 136/30 file after the snapshot 2024-03-28T15:03:24.351 INFO:tasks.workunit.client.0.smithi186.stdout: create 136/31 file after the snapshot 2024-03-28T15:03:24.353 INFO:tasks.workunit.client.0.smithi186.stdout: create 136/32 file after the snapshot 2024-03-28T15:03:24.355 INFO:tasks.workunit.client.0.smithi186.stdout: create 136/33 file after the snapshot 2024-03-28T15:03:24.356 INFO:tasks.workunit.client.0.smithi186.stdout: create 136/34 file after the snapshot 2024-03-28T15:03:24.358 INFO:tasks.workunit.client.0.smithi186.stdout: create 136/35 file after the snapshot 2024-03-28T15:03:24.360 INFO:tasks.workunit.client.0.smithi186.stdout: create 136/36 file after the snapshot 2024-03-28T15:03:24.362 INFO:tasks.workunit.client.0.smithi186.stdout: create 136/37 file after the snapshot 2024-03-28T15:03:24.363 INFO:tasks.workunit.client.0.smithi186.stdout: create 136/38 file after the snapshot 2024-03-28T15:03:24.365 INFO:tasks.workunit.client.0.smithi186.stdout: create 136/39 file after the snapshot 2024-03-28T15:03:24.367 INFO:tasks.workunit.client.0.smithi186.stdout: create 137/.snap/snap-subdir-test 2024-03-28T15:03:24.387 INFO:tasks.workunit.client.0.smithi186.stdout: create 137/30 file after the snapshot 2024-03-28T15:03:24.388 INFO:tasks.workunit.client.0.smithi186.stdout: create 137/31 file after the snapshot 2024-03-28T15:03:24.390 INFO:tasks.workunit.client.0.smithi186.stdout: create 137/32 file after the snapshot 2024-03-28T15:03:24.392 INFO:tasks.workunit.client.0.smithi186.stdout: create 137/33 file after the snapshot 2024-03-28T15:03:24.394 INFO:tasks.workunit.client.0.smithi186.stdout: create 137/34 file after the snapshot 2024-03-28T15:03:24.396 INFO:tasks.workunit.client.0.smithi186.stdout: create 137/35 file after the snapshot 2024-03-28T15:03:24.397 INFO:tasks.workunit.client.0.smithi186.stdout: create 137/36 file after the snapshot 2024-03-28T15:03:24.399 INFO:tasks.workunit.client.0.smithi186.stdout: create 137/37 file after the snapshot 2024-03-28T15:03:24.400 INFO:tasks.workunit.client.0.smithi186.stdout: create 137/38 file after the snapshot 2024-03-28T15:03:24.402 INFO:tasks.workunit.client.0.smithi186.stdout: create 137/39 file after the snapshot 2024-03-28T15:03:24.404 INFO:tasks.workunit.client.0.smithi186.stdout: create 138/.snap/snap-subdir-test 2024-03-28T15:03:24.423 INFO:tasks.workunit.client.0.smithi186.stdout: create 138/30 file after the snapshot 2024-03-28T15:03:24.424 INFO:tasks.workunit.client.0.smithi186.stdout: create 138/31 file after the snapshot 2024-03-28T15:03:24.426 INFO:tasks.workunit.client.0.smithi186.stdout: create 138/32 file after the snapshot 2024-03-28T15:03:24.428 INFO:tasks.workunit.client.0.smithi186.stdout: create 138/33 file after the snapshot 2024-03-28T15:03:24.430 INFO:tasks.workunit.client.0.smithi186.stdout: create 138/34 file after the snapshot 2024-03-28T15:03:24.432 INFO:tasks.workunit.client.0.smithi186.stdout: create 138/35 file after the snapshot 2024-03-28T15:03:24.433 INFO:tasks.workunit.client.0.smithi186.stdout: create 138/36 file after the snapshot 2024-03-28T15:03:24.435 INFO:tasks.workunit.client.0.smithi186.stdout: create 138/37 file after the snapshot 2024-03-28T15:03:24.437 INFO:tasks.workunit.client.0.smithi186.stdout: create 138/38 file after the snapshot 2024-03-28T15:03:24.438 INFO:tasks.workunit.client.0.smithi186.stdout: create 138/39 file after the snapshot 2024-03-28T15:03:24.440 INFO:tasks.workunit.client.0.smithi186.stdout: create 139/.snap/snap-subdir-test 2024-03-28T15:03:24.467 INFO:tasks.workunit.client.0.smithi186.stdout: create 139/30 file after the snapshot 2024-03-28T15:03:24.468 INFO:tasks.workunit.client.0.smithi186.stdout: create 139/31 file after the snapshot 2024-03-28T15:03:24.470 INFO:tasks.workunit.client.0.smithi186.stdout: create 139/32 file after the snapshot 2024-03-28T15:03:24.471 INFO:tasks.workunit.client.0.smithi186.stdout: create 139/33 file after the snapshot 2024-03-28T15:03:24.473 INFO:tasks.workunit.client.0.smithi186.stdout: create 139/34 file after the snapshot 2024-03-28T15:03:24.475 INFO:tasks.workunit.client.0.smithi186.stdout: create 139/35 file after the snapshot 2024-03-28T15:03:24.477 INFO:tasks.workunit.client.0.smithi186.stdout: create 139/36 file after the snapshot 2024-03-28T15:03:24.479 INFO:tasks.workunit.client.0.smithi186.stdout: create 139/37 file after the snapshot 2024-03-28T15:03:24.481 INFO:tasks.workunit.client.0.smithi186.stdout: create 139/38 file after the snapshot 2024-03-28T15:03:24.483 INFO:tasks.workunit.client.0.smithi186.stdout: create 139/39 file after the snapshot 2024-03-28T15:03:24.485 INFO:tasks.workunit.client.0.smithi186.stdout: create 140/.snap/snap-subdir-test 2024-03-28T15:03:24.511 INFO:tasks.workunit.client.0.smithi186.stdout: create 140/30 file after the snapshot 2024-03-28T15:03:24.513 INFO:tasks.workunit.client.0.smithi186.stdout: create 140/31 file after the snapshot 2024-03-28T15:03:24.514 INFO:tasks.workunit.client.0.smithi186.stdout: create 140/32 file after the snapshot 2024-03-28T15:03:24.516 INFO:tasks.workunit.client.0.smithi186.stdout: create 140/33 file after the snapshot 2024-03-28T15:03:24.518 INFO:tasks.workunit.client.0.smithi186.stdout: create 140/34 file after the snapshot 2024-03-28T15:03:24.520 INFO:tasks.workunit.client.0.smithi186.stdout: create 140/35 file after the snapshot 2024-03-28T15:03:24.521 INFO:tasks.workunit.client.0.smithi186.stdout: create 140/36 file after the snapshot 2024-03-28T15:03:24.523 INFO:tasks.workunit.client.0.smithi186.stdout: create 140/37 file after the snapshot 2024-03-28T15:03:24.525 INFO:tasks.workunit.client.0.smithi186.stdout: create 140/38 file after the snapshot 2024-03-28T15:03:24.527 INFO:tasks.workunit.client.0.smithi186.stdout: create 140/39 file after the snapshot 2024-03-28T15:03:24.528 INFO:tasks.workunit.client.0.smithi186.stdout: create 141/.snap/snap-subdir-test 2024-03-28T15:03:24.547 INFO:tasks.workunit.client.0.smithi186.stdout: create 141/30 file after the snapshot 2024-03-28T15:03:24.549 INFO:tasks.workunit.client.0.smithi186.stdout: create 141/31 file after the snapshot 2024-03-28T15:03:24.550 INFO:tasks.workunit.client.0.smithi186.stdout: create 141/32 file after the snapshot 2024-03-28T15:03:24.552 INFO:tasks.workunit.client.0.smithi186.stdout: create 141/33 file after the snapshot 2024-03-28T15:03:24.554 INFO:tasks.workunit.client.0.smithi186.stdout: create 141/34 file after the snapshot 2024-03-28T15:03:24.556 INFO:tasks.workunit.client.0.smithi186.stdout: create 141/35 file after the snapshot 2024-03-28T15:03:24.557 INFO:tasks.workunit.client.0.smithi186.stdout: create 141/36 file after the snapshot 2024-03-28T15:03:24.559 INFO:tasks.workunit.client.0.smithi186.stdout: create 141/37 file after the snapshot 2024-03-28T15:03:24.561 INFO:tasks.workunit.client.0.smithi186.stdout: create 141/38 file after the snapshot 2024-03-28T15:03:24.562 INFO:tasks.workunit.client.0.smithi186.stdout: create 141/39 file after the snapshot 2024-03-28T15:03:24.564 INFO:tasks.workunit.client.0.smithi186.stdout: create 142/.snap/snap-subdir-test 2024-03-28T15:03:24.590 INFO:tasks.workunit.client.0.smithi186.stdout: create 142/30 file after the snapshot 2024-03-28T15:03:24.592 INFO:tasks.workunit.client.0.smithi186.stdout: create 142/31 file after the snapshot 2024-03-28T15:03:24.593 INFO:tasks.workunit.client.0.smithi186.stdout: create 142/32 file after the snapshot 2024-03-28T15:03:24.595 INFO:tasks.workunit.client.0.smithi186.stdout: create 142/33 file after the snapshot 2024-03-28T15:03:24.597 INFO:tasks.workunit.client.0.smithi186.stdout: create 142/34 file after the snapshot 2024-03-28T15:03:24.598 INFO:tasks.workunit.client.0.smithi186.stdout: create 142/35 file after the snapshot 2024-03-28T15:03:24.600 INFO:tasks.workunit.client.0.smithi186.stdout: create 142/36 file after the snapshot 2024-03-28T15:03:24.602 INFO:tasks.workunit.client.0.smithi186.stdout: create 142/37 file after the snapshot 2024-03-28T15:03:24.603 INFO:tasks.workunit.client.0.smithi186.stdout: create 142/38 file after the snapshot 2024-03-28T15:03:24.605 INFO:tasks.workunit.client.0.smithi186.stdout: create 142/39 file after the snapshot 2024-03-28T15:03:24.607 INFO:tasks.workunit.client.0.smithi186.stdout: create 143/.snap/snap-subdir-test 2024-03-28T15:03:24.627 INFO:tasks.workunit.client.0.smithi186.stdout: create 143/30 file after the snapshot 2024-03-28T15:03:24.629 INFO:tasks.workunit.client.0.smithi186.stdout: create 143/31 file after the snapshot 2024-03-28T15:03:24.631 INFO:tasks.workunit.client.0.smithi186.stdout: create 143/32 file after the snapshot 2024-03-28T15:03:24.633 INFO:tasks.workunit.client.0.smithi186.stdout: create 143/33 file after the snapshot 2024-03-28T15:03:24.635 INFO:tasks.workunit.client.0.smithi186.stdout: create 143/34 file after the snapshot 2024-03-28T15:03:24.637 INFO:tasks.workunit.client.0.smithi186.stdout: create 143/35 file after the snapshot 2024-03-28T15:03:24.638 INFO:tasks.workunit.client.0.smithi186.stdout: create 143/36 file after the snapshot 2024-03-28T15:03:24.640 INFO:tasks.workunit.client.0.smithi186.stdout: create 143/37 file after the snapshot 2024-03-28T15:03:24.642 INFO:tasks.workunit.client.0.smithi186.stdout: create 143/38 file after the snapshot 2024-03-28T15:03:24.644 INFO:tasks.workunit.client.0.smithi186.stdout: create 143/39 file after the snapshot 2024-03-28T15:03:24.646 INFO:tasks.workunit.client.0.smithi186.stdout: create 144/.snap/snap-subdir-test 2024-03-28T15:03:24.665 INFO:tasks.workunit.client.0.smithi186.stdout: create 144/30 file after the snapshot 2024-03-28T15:03:24.667 INFO:tasks.workunit.client.0.smithi186.stdout: create 144/31 file after the snapshot 2024-03-28T15:03:24.669 INFO:tasks.workunit.client.0.smithi186.stdout: create 144/32 file after the snapshot 2024-03-28T15:03:24.672 INFO:tasks.workunit.client.0.smithi186.stdout: create 144/33 file after the snapshot 2024-03-28T15:03:24.673 INFO:tasks.workunit.client.0.smithi186.stdout: create 144/34 file after the snapshot 2024-03-28T15:03:24.675 INFO:tasks.workunit.client.0.smithi186.stdout: create 144/35 file after the snapshot 2024-03-28T15:03:24.677 INFO:tasks.workunit.client.0.smithi186.stdout: create 144/36 file after the snapshot 2024-03-28T15:03:24.679 INFO:tasks.workunit.client.0.smithi186.stdout: create 144/37 file after the snapshot 2024-03-28T15:03:24.681 INFO:tasks.workunit.client.0.smithi186.stdout: create 144/38 file after the snapshot 2024-03-28T15:03:24.682 INFO:tasks.workunit.client.0.smithi186.stdout: create 144/39 file after the snapshot 2024-03-28T15:03:24.684 INFO:tasks.workunit.client.0.smithi186.stdout: create 145/.snap/snap-subdir-test 2024-03-28T15:03:24.702 INFO:tasks.workunit.client.0.smithi186.stdout: create 145/30 file after the snapshot 2024-03-28T15:03:24.704 INFO:tasks.workunit.client.0.smithi186.stdout: create 145/31 file after the snapshot 2024-03-28T15:03:24.706 INFO:tasks.workunit.client.0.smithi186.stdout: create 145/32 file after the snapshot 2024-03-28T15:03:24.707 INFO:tasks.workunit.client.0.smithi186.stdout: create 145/33 file after the snapshot 2024-03-28T15:03:24.709 INFO:tasks.workunit.client.0.smithi186.stdout: create 145/34 file after the snapshot 2024-03-28T15:03:24.711 INFO:tasks.workunit.client.0.smithi186.stdout: create 145/35 file after the snapshot 2024-03-28T15:03:24.713 INFO:tasks.workunit.client.0.smithi186.stdout: create 145/36 file after the snapshot 2024-03-28T15:03:24.715 INFO:tasks.workunit.client.0.smithi186.stdout: create 145/37 file after the snapshot 2024-03-28T15:03:24.717 INFO:tasks.workunit.client.0.smithi186.stdout: create 145/38 file after the snapshot 2024-03-28T15:03:24.718 INFO:tasks.workunit.client.0.smithi186.stdout: create 145/39 file after the snapshot 2024-03-28T15:03:24.720 INFO:tasks.workunit.client.0.smithi186.stdout: create 146/.snap/snap-subdir-test 2024-03-28T15:03:24.746 INFO:tasks.workunit.client.0.smithi186.stdout: create 146/30 file after the snapshot 2024-03-28T15:03:24.748 INFO:tasks.workunit.client.0.smithi186.stdout: create 146/31 file after the snapshot 2024-03-28T15:03:24.750 INFO:tasks.workunit.client.0.smithi186.stdout: create 146/32 file after the snapshot 2024-03-28T15:03:24.751 INFO:tasks.workunit.client.0.smithi186.stdout: create 146/33 file after the snapshot 2024-03-28T15:03:24.753 INFO:tasks.workunit.client.0.smithi186.stdout: create 146/34 file after the snapshot 2024-03-28T15:03:24.754 INFO:tasks.workunit.client.0.smithi186.stdout: create 146/35 file after the snapshot 2024-03-28T15:03:24.756 INFO:tasks.workunit.client.0.smithi186.stdout: create 146/36 file after the snapshot 2024-03-28T15:03:24.758 INFO:tasks.workunit.client.0.smithi186.stdout: create 146/37 file after the snapshot 2024-03-28T15:03:24.759 INFO:tasks.workunit.client.0.smithi186.stdout: create 146/38 file after the snapshot 2024-03-28T15:03:24.761 INFO:tasks.workunit.client.0.smithi186.stdout: create 146/39 file after the snapshot 2024-03-28T15:03:24.763 INFO:tasks.workunit.client.0.smithi186.stdout: create 147/.snap/snap-subdir-test 2024-03-28T15:03:24.782 INFO:tasks.workunit.client.0.smithi186.stdout: create 147/30 file after the snapshot 2024-03-28T15:03:24.784 INFO:tasks.workunit.client.0.smithi186.stdout: create 147/31 file after the snapshot 2024-03-28T15:03:24.786 INFO:tasks.workunit.client.0.smithi186.stdout: create 147/32 file after the snapshot 2024-03-28T15:03:24.788 INFO:tasks.workunit.client.0.smithi186.stdout: create 147/33 file after the snapshot 2024-03-28T15:03:24.790 INFO:tasks.workunit.client.0.smithi186.stdout: create 147/34 file after the snapshot 2024-03-28T15:03:24.792 INFO:tasks.workunit.client.0.smithi186.stdout: create 147/35 file after the snapshot 2024-03-28T15:03:24.794 INFO:tasks.workunit.client.0.smithi186.stdout: create 147/36 file after the snapshot 2024-03-28T15:03:24.824 INFO:tasks.workunit.client.0.smithi186.stdout: create 147/37 file after the snapshot 2024-03-28T15:03:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: create 147/38 file after the snapshot 2024-03-28T15:03:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: create 147/39 file after the snapshot 2024-03-28T15:03:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: create 148/.snap/snap-subdir-test 2024-03-28T15:03:25.059 INFO:tasks.workunit.client.0.smithi186.stdout: create 148/30 file after the snapshot 2024-03-28T15:03:25.060 INFO:tasks.workunit.client.0.smithi186.stdout: create 148/31 file after the snapshot 2024-03-28T15:03:25.062 INFO:tasks.workunit.client.0.smithi186.stdout: create 148/32 file after the snapshot 2024-03-28T15:03:25.063 INFO:tasks.workunit.client.0.smithi186.stdout: create 148/33 file after the snapshot 2024-03-28T15:03:25.065 INFO:tasks.workunit.client.0.smithi186.stdout: create 148/34 file after the snapshot 2024-03-28T15:03:25.067 INFO:tasks.workunit.client.0.smithi186.stdout: create 148/35 file after the snapshot 2024-03-28T15:03:25.069 INFO:tasks.workunit.client.0.smithi186.stdout: create 148/36 file after the snapshot 2024-03-28T15:03:25.070 INFO:tasks.workunit.client.0.smithi186.stdout: create 148/37 file after the snapshot 2024-03-28T15:03:25.072 INFO:tasks.workunit.client.0.smithi186.stdout: create 148/38 file after the snapshot 2024-03-28T15:03:25.074 INFO:tasks.workunit.client.0.smithi186.stdout: create 148/39 file after the snapshot 2024-03-28T15:03:25.076 INFO:tasks.workunit.client.0.smithi186.stdout: create 149/.snap/snap-subdir-test 2024-03-28T15:03:25.093 INFO:tasks.workunit.client.0.smithi186.stdout: create 149/30 file after the snapshot 2024-03-28T15:03:25.094 INFO:tasks.workunit.client.0.smithi186.stdout: create 149/31 file after the snapshot 2024-03-28T15:03:25.096 INFO:tasks.workunit.client.0.smithi186.stdout: create 149/32 file after the snapshot 2024-03-28T15:03:25.097 INFO:tasks.workunit.client.0.smithi186.stdout: create 149/33 file after the snapshot 2024-03-28T15:03:25.099 INFO:tasks.workunit.client.0.smithi186.stdout: create 149/34 file after the snapshot 2024-03-28T15:03:25.101 INFO:tasks.workunit.client.0.smithi186.stdout: create 149/35 file after the snapshot 2024-03-28T15:03:25.102 INFO:tasks.workunit.client.0.smithi186.stdout: create 149/36 file after the snapshot 2024-03-28T15:03:25.104 INFO:tasks.workunit.client.0.smithi186.stdout: create 149/37 file after the snapshot 2024-03-28T15:03:25.106 INFO:tasks.workunit.client.0.smithi186.stdout: create 149/38 file after the snapshot 2024-03-28T15:03:25.110 INFO:tasks.workunit.client.0.smithi186.stdout: create 149/39 file after the snapshot 2024-03-28T15:03:25.110 INFO:tasks.workunit.client.0.smithi186.stdout: create 150/.snap/snap-subdir-test 2024-03-28T15:03:25.130 INFO:tasks.workunit.client.0.smithi186.stdout: create 150/30 file after the snapshot 2024-03-28T15:03:25.132 INFO:tasks.workunit.client.0.smithi186.stdout: create 150/31 file after the snapshot 2024-03-28T15:03:25.134 INFO:tasks.workunit.client.0.smithi186.stdout: create 150/32 file after the snapshot 2024-03-28T15:03:25.135 INFO:tasks.workunit.client.0.smithi186.stdout: create 150/33 file after the snapshot 2024-03-28T15:03:25.137 INFO:tasks.workunit.client.0.smithi186.stdout: create 150/34 file after the snapshot 2024-03-28T15:03:25.139 INFO:tasks.workunit.client.0.smithi186.stdout: create 150/35 file after the snapshot 2024-03-28T15:03:25.140 INFO:tasks.workunit.client.0.smithi186.stdout: create 150/36 file after the snapshot 2024-03-28T15:03:25.142 INFO:tasks.workunit.client.0.smithi186.stdout: create 150/37 file after the snapshot 2024-03-28T15:03:25.144 INFO:tasks.workunit.client.0.smithi186.stdout: create 150/38 file after the snapshot 2024-03-28T15:03:25.146 INFO:tasks.workunit.client.0.smithi186.stdout: create 150/39 file after the snapshot 2024-03-28T15:03:25.147 INFO:tasks.workunit.client.0.smithi186.stdout: create 151/.snap/snap-subdir-test 2024-03-28T15:03:25.166 INFO:tasks.workunit.client.0.smithi186.stdout: create 151/30 file after the snapshot 2024-03-28T15:03:25.168 INFO:tasks.workunit.client.0.smithi186.stdout: create 151/31 file after the snapshot 2024-03-28T15:03:25.169 INFO:tasks.workunit.client.0.smithi186.stdout: create 151/32 file after the snapshot 2024-03-28T15:03:25.171 INFO:tasks.workunit.client.0.smithi186.stdout: create 151/33 file after the snapshot 2024-03-28T15:03:25.173 INFO:tasks.workunit.client.0.smithi186.stdout: create 151/34 file after the snapshot 2024-03-28T15:03:25.175 INFO:tasks.workunit.client.0.smithi186.stdout: create 151/35 file after the snapshot 2024-03-28T15:03:25.176 INFO:tasks.workunit.client.0.smithi186.stdout: create 151/36 file after the snapshot 2024-03-28T15:03:25.178 INFO:tasks.workunit.client.0.smithi186.stdout: create 151/37 file after the snapshot 2024-03-28T15:03:25.180 INFO:tasks.workunit.client.0.smithi186.stdout: create 151/38 file after the snapshot 2024-03-28T15:03:25.181 INFO:tasks.workunit.client.0.smithi186.stdout: create 151/39 file after the snapshot 2024-03-28T15:03:25.183 INFO:tasks.workunit.client.0.smithi186.stdout: create 152/.snap/snap-subdir-test 2024-03-28T15:03:25.201 INFO:tasks.workunit.client.0.smithi186.stdout: create 152/30 file after the snapshot 2024-03-28T15:03:25.203 INFO:tasks.workunit.client.0.smithi186.stdout: create 152/31 file after the snapshot 2024-03-28T15:03:25.205 INFO:tasks.workunit.client.0.smithi186.stdout: create 152/32 file after the snapshot 2024-03-28T15:03:25.206 INFO:tasks.workunit.client.0.smithi186.stdout: create 152/33 file after the snapshot 2024-03-28T15:03:25.208 INFO:tasks.workunit.client.0.smithi186.stdout: create 152/34 file after the snapshot 2024-03-28T15:03:25.210 INFO:tasks.workunit.client.0.smithi186.stdout: create 152/35 file after the snapshot 2024-03-28T15:03:25.212 INFO:tasks.workunit.client.0.smithi186.stdout: create 152/36 file after the snapshot 2024-03-28T15:03:25.213 INFO:tasks.workunit.client.0.smithi186.stdout: create 152/37 file after the snapshot 2024-03-28T15:03:25.215 INFO:tasks.workunit.client.0.smithi186.stdout: create 152/38 file after the snapshot 2024-03-28T15:03:25.217 INFO:tasks.workunit.client.0.smithi186.stdout: create 152/39 file after the snapshot 2024-03-28T15:03:25.218 INFO:tasks.workunit.client.0.smithi186.stdout: create 153/.snap/snap-subdir-test 2024-03-28T15:03:25.235 INFO:tasks.workunit.client.0.smithi186.stdout: create 153/30 file after the snapshot 2024-03-28T15:03:25.236 INFO:tasks.workunit.client.0.smithi186.stdout: create 153/31 file after the snapshot 2024-03-28T15:03:25.238 INFO:tasks.workunit.client.0.smithi186.stdout: create 153/32 file after the snapshot 2024-03-28T15:03:25.240 INFO:tasks.workunit.client.0.smithi186.stdout: create 153/33 file after the snapshot 2024-03-28T15:03:25.241 INFO:tasks.workunit.client.0.smithi186.stdout: create 153/34 file after the snapshot 2024-03-28T15:03:25.243 INFO:tasks.workunit.client.0.smithi186.stdout: create 153/35 file after the snapshot 2024-03-28T15:03:25.245 INFO:tasks.workunit.client.0.smithi186.stdout: create 153/36 file after the snapshot 2024-03-28T15:03:25.246 INFO:tasks.workunit.client.0.smithi186.stdout: create 153/37 file after the snapshot 2024-03-28T15:03:25.248 INFO:tasks.workunit.client.0.smithi186.stdout: create 153/38 file after the snapshot 2024-03-28T15:03:25.250 INFO:tasks.workunit.client.0.smithi186.stdout: create 153/39 file after the snapshot 2024-03-28T15:03:25.252 INFO:tasks.workunit.client.0.smithi186.stdout: create 154/.snap/snap-subdir-test 2024-03-28T15:03:25.277 INFO:tasks.workunit.client.0.smithi186.stdout: create 154/30 file after the snapshot 2024-03-28T15:03:25.279 INFO:tasks.workunit.client.0.smithi186.stdout: create 154/31 file after the snapshot 2024-03-28T15:03:25.281 INFO:tasks.workunit.client.0.smithi186.stdout: create 154/32 file after the snapshot 2024-03-28T15:03:25.283 INFO:tasks.workunit.client.0.smithi186.stdout: create 154/33 file after the snapshot 2024-03-28T15:03:25.285 INFO:tasks.workunit.client.0.smithi186.stdout: create 154/34 file after the snapshot 2024-03-28T15:03:25.286 INFO:tasks.workunit.client.0.smithi186.stdout: create 154/35 file after the snapshot 2024-03-28T15:03:25.288 INFO:tasks.workunit.client.0.smithi186.stdout: create 154/36 file after the snapshot 2024-03-28T15:03:25.291 INFO:tasks.workunit.client.0.smithi186.stdout: create 154/37 file after the snapshot 2024-03-28T15:03:25.292 INFO:tasks.workunit.client.0.smithi186.stdout: create 154/38 file after the snapshot 2024-03-28T15:03:25.294 INFO:tasks.workunit.client.0.smithi186.stdout: create 154/39 file after the snapshot 2024-03-28T15:03:25.296 INFO:tasks.workunit.client.0.smithi186.stdout: create 155/.snap/snap-subdir-test 2024-03-28T15:03:25.323 INFO:tasks.workunit.client.0.smithi186.stdout: create 155/30 file after the snapshot 2024-03-28T15:03:25.324 INFO:tasks.workunit.client.0.smithi186.stdout: create 155/31 file after the snapshot 2024-03-28T15:03:25.326 INFO:tasks.workunit.client.0.smithi186.stdout: create 155/32 file after the snapshot 2024-03-28T15:03:25.328 INFO:tasks.workunit.client.0.smithi186.stdout: create 155/33 file after the snapshot 2024-03-28T15:03:25.330 INFO:tasks.workunit.client.0.smithi186.stdout: create 155/34 file after the snapshot 2024-03-28T15:03:25.331 INFO:tasks.workunit.client.0.smithi186.stdout: create 155/35 file after the snapshot 2024-03-28T15:03:25.333 INFO:tasks.workunit.client.0.smithi186.stdout: create 155/36 file after the snapshot 2024-03-28T15:03:25.335 INFO:tasks.workunit.client.0.smithi186.stdout: create 155/37 file after the snapshot 2024-03-28T15:03:25.336 INFO:tasks.workunit.client.0.smithi186.stdout: create 155/38 file after the snapshot 2024-03-28T15:03:25.338 INFO:tasks.workunit.client.0.smithi186.stdout: create 155/39 file after the snapshot 2024-03-28T15:03:25.340 INFO:tasks.workunit.client.0.smithi186.stdout: create 156/.snap/snap-subdir-test 2024-03-28T15:03:25.361 INFO:tasks.workunit.client.0.smithi186.stdout: create 156/30 file after the snapshot 2024-03-28T15:03:25.363 INFO:tasks.workunit.client.0.smithi186.stdout: create 156/31 file after the snapshot 2024-03-28T15:03:25.364 INFO:tasks.workunit.client.0.smithi186.stdout: create 156/32 file after the snapshot 2024-03-28T15:03:25.366 INFO:tasks.workunit.client.0.smithi186.stdout: create 156/33 file after the snapshot 2024-03-28T15:03:25.368 INFO:tasks.workunit.client.0.smithi186.stdout: create 156/34 file after the snapshot 2024-03-28T15:03:25.369 INFO:tasks.workunit.client.0.smithi186.stdout: create 156/35 file after the snapshot 2024-03-28T15:03:25.371 INFO:tasks.workunit.client.0.smithi186.stdout: create 156/36 file after the snapshot 2024-03-28T15:03:25.373 INFO:tasks.workunit.client.0.smithi186.stdout: create 156/37 file after the snapshot 2024-03-28T15:03:25.375 INFO:tasks.workunit.client.0.smithi186.stdout: create 156/38 file after the snapshot 2024-03-28T15:03:25.377 INFO:tasks.workunit.client.0.smithi186.stdout: create 156/39 file after the snapshot 2024-03-28T15:03:25.378 INFO:tasks.workunit.client.0.smithi186.stdout: create 157/.snap/snap-subdir-test 2024-03-28T15:03:25.403 INFO:tasks.workunit.client.0.smithi186.stdout: create 157/30 file after the snapshot 2024-03-28T15:03:25.404 INFO:tasks.workunit.client.0.smithi186.stdout: create 157/31 file after the snapshot 2024-03-28T15:03:25.406 INFO:tasks.workunit.client.0.smithi186.stdout: create 157/32 file after the snapshot 2024-03-28T15:03:25.408 INFO:tasks.workunit.client.0.smithi186.stdout: create 157/33 file after the snapshot 2024-03-28T15:03:25.410 INFO:tasks.workunit.client.0.smithi186.stdout: create 157/34 file after the snapshot 2024-03-28T15:03:25.412 INFO:tasks.workunit.client.0.smithi186.stdout: create 157/35 file after the snapshot 2024-03-28T15:03:25.414 INFO:tasks.workunit.client.0.smithi186.stdout: create 157/36 file after the snapshot 2024-03-28T15:03:25.416 INFO:tasks.workunit.client.0.smithi186.stdout: create 157/37 file after the snapshot 2024-03-28T15:03:25.417 INFO:tasks.workunit.client.0.smithi186.stdout: create 157/38 file after the snapshot 2024-03-28T15:03:25.419 INFO:tasks.workunit.client.0.smithi186.stdout: create 157/39 file after the snapshot 2024-03-28T15:03:25.421 INFO:tasks.workunit.client.0.smithi186.stdout: create 158/.snap/snap-subdir-test 2024-03-28T15:03:25.438 INFO:tasks.workunit.client.0.smithi186.stdout: create 158/30 file after the snapshot 2024-03-28T15:03:25.440 INFO:tasks.workunit.client.0.smithi186.stdout: create 158/31 file after the snapshot 2024-03-28T15:03:25.441 INFO:tasks.workunit.client.0.smithi186.stdout: create 158/32 file after the snapshot 2024-03-28T15:03:25.443 INFO:tasks.workunit.client.0.smithi186.stdout: create 158/33 file after the snapshot 2024-03-28T15:03:25.445 INFO:tasks.workunit.client.0.smithi186.stdout: create 158/34 file after the snapshot 2024-03-28T15:03:25.446 INFO:tasks.workunit.client.0.smithi186.stdout: create 158/35 file after the snapshot 2024-03-28T15:03:25.449 INFO:tasks.workunit.client.0.smithi186.stdout: create 158/36 file after the snapshot 2024-03-28T15:03:25.450 INFO:tasks.workunit.client.0.smithi186.stdout: create 158/37 file after the snapshot 2024-03-28T15:03:25.452 INFO:tasks.workunit.client.0.smithi186.stdout: create 158/38 file after the snapshot 2024-03-28T15:03:25.453 INFO:tasks.workunit.client.0.smithi186.stdout: create 158/39 file after the snapshot 2024-03-28T15:03:25.455 INFO:tasks.workunit.client.0.smithi186.stdout: create 159/.snap/snap-subdir-test 2024-03-28T15:03:25.474 INFO:tasks.workunit.client.0.smithi186.stdout: create 159/30 file after the snapshot 2024-03-28T15:03:25.476 INFO:tasks.workunit.client.0.smithi186.stdout: create 159/31 file after the snapshot 2024-03-28T15:03:25.478 INFO:tasks.workunit.client.0.smithi186.stdout: create 159/32 file after the snapshot 2024-03-28T15:03:25.480 INFO:tasks.workunit.client.0.smithi186.stdout: create 159/33 file after the snapshot 2024-03-28T15:03:25.481 INFO:tasks.workunit.client.0.smithi186.stdout: create 159/34 file after the snapshot 2024-03-28T15:03:25.483 INFO:tasks.workunit.client.0.smithi186.stdout: create 159/35 file after the snapshot 2024-03-28T15:03:25.485 INFO:tasks.workunit.client.0.smithi186.stdout: create 159/36 file after the snapshot 2024-03-28T15:03:25.486 INFO:tasks.workunit.client.0.smithi186.stdout: create 159/37 file after the snapshot 2024-03-28T15:03:25.488 INFO:tasks.workunit.client.0.smithi186.stdout: create 159/38 file after the snapshot 2024-03-28T15:03:25.490 INFO:tasks.workunit.client.0.smithi186.stdout: create 159/39 file after the snapshot 2024-03-28T15:03:25.492 INFO:tasks.workunit.client.0.smithi186.stdout: create 160/.snap/snap-subdir-test 2024-03-28T15:03:25.517 INFO:tasks.workunit.client.0.smithi186.stdout: create 160/30 file after the snapshot 2024-03-28T15:03:25.518 INFO:tasks.workunit.client.0.smithi186.stdout: create 160/31 file after the snapshot 2024-03-28T15:03:25.520 INFO:tasks.workunit.client.0.smithi186.stdout: create 160/32 file after the snapshot 2024-03-28T15:03:25.520 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:03:25.521 INFO:tasks.workunit.client.0.smithi186.stdout: create 160/33 file after the snapshot 2024-03-28T15:03:25.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:03:25.524 INFO:tasks.workunit.client.0.smithi186.stdout: create 160/34 file after the snapshot 2024-03-28T15:03:25.526 INFO:tasks.workunit.client.0.smithi186.stdout: create 160/35 file after the snapshot 2024-03-28T15:03:25.527 INFO:tasks.workunit.client.0.smithi186.stdout: create 160/36 file after the snapshot 2024-03-28T15:03:25.529 INFO:tasks.workunit.client.0.smithi186.stdout: create 160/37 file after the snapshot 2024-03-28T15:03:25.531 INFO:tasks.workunit.client.0.smithi186.stdout: create 160/38 file after the snapshot 2024-03-28T15:03:25.533 INFO:tasks.workunit.client.0.smithi186.stdout: create 160/39 file after the snapshot 2024-03-28T15:03:25.535 INFO:tasks.workunit.client.0.smithi186.stdout: create 161/.snap/snap-subdir-test 2024-03-28T15:03:25.556 INFO:tasks.workunit.client.0.smithi186.stdout: create 161/30 file after the snapshot 2024-03-28T15:03:25.557 INFO:tasks.workunit.client.0.smithi186.stdout: create 161/31 file after the snapshot 2024-03-28T15:03:25.559 INFO:tasks.workunit.client.0.smithi186.stdout: create 161/32 file after the snapshot 2024-03-28T15:03:25.561 INFO:tasks.workunit.client.0.smithi186.stdout: create 161/33 file after the snapshot 2024-03-28T15:03:25.563 INFO:tasks.workunit.client.0.smithi186.stdout: create 161/34 file after the snapshot 2024-03-28T15:03:25.565 INFO:tasks.workunit.client.0.smithi186.stdout: create 161/35 file after the snapshot 2024-03-28T15:03:25.567 INFO:tasks.workunit.client.0.smithi186.stdout: create 161/36 file after the snapshot 2024-03-28T15:03:25.569 INFO:tasks.workunit.client.0.smithi186.stdout: create 161/37 file after the snapshot 2024-03-28T15:03:25.572 INFO:tasks.workunit.client.0.smithi186.stdout: create 161/38 file after the snapshot 2024-03-28T15:03:25.572 INFO:tasks.workunit.client.0.smithi186.stdout: create 161/39 file after the snapshot 2024-03-28T15:03:25.574 INFO:tasks.workunit.client.0.smithi186.stdout: create 162/.snap/snap-subdir-test 2024-03-28T15:03:25.593 INFO:tasks.workunit.client.0.smithi186.stdout: create 162/30 file after the snapshot 2024-03-28T15:03:25.595 INFO:tasks.workunit.client.0.smithi186.stdout: create 162/31 file after the snapshot 2024-03-28T15:03:25.596 INFO:tasks.workunit.client.0.smithi186.stdout: create 162/32 file after the snapshot 2024-03-28T15:03:25.598 INFO:tasks.workunit.client.0.smithi186.stdout: create 162/33 file after the snapshot 2024-03-28T15:03:25.600 INFO:tasks.workunit.client.0.smithi186.stdout: create 162/34 file after the snapshot 2024-03-28T15:03:25.602 INFO:tasks.workunit.client.0.smithi186.stdout: create 162/35 file after the snapshot 2024-03-28T15:03:25.603 INFO:tasks.workunit.client.0.smithi186.stdout: create 162/36 file after the snapshot 2024-03-28T15:03:25.605 INFO:tasks.workunit.client.0.smithi186.stdout: create 162/37 file after the snapshot 2024-03-28T15:03:25.607 INFO:tasks.workunit.client.0.smithi186.stdout: create 162/38 file after the snapshot 2024-03-28T15:03:25.609 INFO:tasks.workunit.client.0.smithi186.stdout: create 162/39 file after the snapshot 2024-03-28T15:03:25.611 INFO:tasks.workunit.client.0.smithi186.stdout: create 163/.snap/snap-subdir-test 2024-03-28T15:03:25.637 INFO:tasks.workunit.client.0.smithi186.stdout: create 163/30 file after the snapshot 2024-03-28T15:03:25.639 INFO:tasks.workunit.client.0.smithi186.stdout: create 163/31 file after the snapshot 2024-03-28T15:03:25.641 INFO:tasks.workunit.client.0.smithi186.stdout: create 163/32 file after the snapshot 2024-03-28T15:03:25.643 INFO:tasks.workunit.client.0.smithi186.stdout: create 163/33 file after the snapshot 2024-03-28T15:03:25.645 INFO:tasks.workunit.client.0.smithi186.stdout: create 163/34 file after the snapshot 2024-03-28T15:03:25.646 INFO:tasks.workunit.client.0.smithi186.stdout: create 163/35 file after the snapshot 2024-03-28T15:03:25.648 INFO:tasks.workunit.client.0.smithi186.stdout: create 163/36 file after the snapshot 2024-03-28T15:03:25.650 INFO:tasks.workunit.client.0.smithi186.stdout: create 163/37 file after the snapshot 2024-03-28T15:03:25.652 INFO:tasks.workunit.client.0.smithi186.stdout: create 163/38 file after the snapshot 2024-03-28T15:03:25.653 INFO:tasks.workunit.client.0.smithi186.stdout: create 163/39 file after the snapshot 2024-03-28T15:03:25.655 INFO:tasks.workunit.client.0.smithi186.stdout: create 164/.snap/snap-subdir-test 2024-03-28T15:03:25.673 INFO:tasks.workunit.client.0.smithi186.stdout: create 164/30 file after the snapshot 2024-03-28T15:03:25.674 INFO:tasks.workunit.client.0.smithi186.stdout: create 164/31 file after the snapshot 2024-03-28T15:03:25.676 INFO:tasks.workunit.client.0.smithi186.stdout: create 164/32 file after the snapshot 2024-03-28T15:03:25.678 INFO:tasks.workunit.client.0.smithi186.stdout: create 164/33 file after the snapshot 2024-03-28T15:03:25.680 INFO:tasks.workunit.client.0.smithi186.stdout: create 164/34 file after the snapshot 2024-03-28T15:03:25.682 INFO:tasks.workunit.client.0.smithi186.stdout: create 164/35 file after the snapshot 2024-03-28T15:03:25.684 INFO:tasks.workunit.client.0.smithi186.stdout: create 164/36 file after the snapshot 2024-03-28T15:03:25.685 INFO:tasks.workunit.client.0.smithi186.stdout: create 164/37 file after the snapshot 2024-03-28T15:03:25.687 INFO:tasks.workunit.client.0.smithi186.stdout: create 164/38 file after the snapshot 2024-03-28T15:03:25.690 INFO:tasks.workunit.client.0.smithi186.stdout: create 164/39 file after the snapshot 2024-03-28T15:03:25.691 INFO:tasks.workunit.client.0.smithi186.stdout: create 165/.snap/snap-subdir-test 2024-03-28T15:03:25.710 INFO:tasks.workunit.client.0.smithi186.stdout: create 165/30 file after the snapshot 2024-03-28T15:03:25.712 INFO:tasks.workunit.client.0.smithi186.stdout: create 165/31 file after the snapshot 2024-03-28T15:03:25.713 INFO:tasks.workunit.client.0.smithi186.stdout: create 165/32 file after the snapshot 2024-03-28T15:03:25.715 INFO:tasks.workunit.client.0.smithi186.stdout: create 165/33 file after the snapshot 2024-03-28T15:03:25.717 INFO:tasks.workunit.client.0.smithi186.stdout: create 165/34 file after the snapshot 2024-03-28T15:03:25.718 INFO:tasks.workunit.client.0.smithi186.stdout: create 165/35 file after the snapshot 2024-03-28T15:03:25.720 INFO:tasks.workunit.client.0.smithi186.stdout: create 165/36 file after the snapshot 2024-03-28T15:03:25.722 INFO:tasks.workunit.client.0.smithi186.stdout: create 165/37 file after the snapshot 2024-03-28T15:03:25.724 INFO:tasks.workunit.client.0.smithi186.stdout: create 165/38 file after the snapshot 2024-03-28T15:03:25.726 INFO:tasks.workunit.client.0.smithi186.stdout: create 165/39 file after the snapshot 2024-03-28T15:03:25.728 INFO:tasks.workunit.client.0.smithi186.stdout: create 166/.snap/snap-subdir-test 2024-03-28T15:03:25.746 INFO:tasks.workunit.client.0.smithi186.stdout: create 166/30 file after the snapshot 2024-03-28T15:03:25.748 INFO:tasks.workunit.client.0.smithi186.stdout: create 166/31 file after the snapshot 2024-03-28T15:03:25.749 INFO:tasks.workunit.client.0.smithi186.stdout: create 166/32 file after the snapshot 2024-03-28T15:03:25.751 INFO:tasks.workunit.client.0.smithi186.stdout: create 166/33 file after the snapshot 2024-03-28T15:03:25.753 INFO:tasks.workunit.client.0.smithi186.stdout: create 166/34 file after the snapshot 2024-03-28T15:03:25.755 INFO:tasks.workunit.client.0.smithi186.stdout: create 166/35 file after the snapshot 2024-03-28T15:03:25.757 INFO:tasks.workunit.client.0.smithi186.stdout: create 166/36 file after the snapshot 2024-03-28T15:03:25.759 INFO:tasks.workunit.client.0.smithi186.stdout: create 166/37 file after the snapshot 2024-03-28T15:03:25.761 INFO:tasks.workunit.client.0.smithi186.stdout: create 166/38 file after the snapshot 2024-03-28T15:03:25.763 INFO:tasks.workunit.client.0.smithi186.stdout: create 166/39 file after the snapshot 2024-03-28T15:03:25.766 INFO:tasks.workunit.client.0.smithi186.stdout: create 167/.snap/snap-subdir-test 2024-03-28T15:03:25.783 INFO:tasks.workunit.client.0.smithi186.stdout: create 167/30 file after the snapshot 2024-03-28T15:03:25.785 INFO:tasks.workunit.client.0.smithi186.stdout: create 167/31 file after the snapshot 2024-03-28T15:03:25.787 INFO:tasks.workunit.client.0.smithi186.stdout: create 167/32 file after the snapshot 2024-03-28T15:03:25.789 INFO:tasks.workunit.client.0.smithi186.stdout: create 167/33 file after the snapshot 2024-03-28T15:03:25.791 INFO:tasks.workunit.client.0.smithi186.stdout: create 167/34 file after the snapshot 2024-03-28T15:03:25.792 INFO:tasks.workunit.client.0.smithi186.stdout: create 167/35 file after the snapshot 2024-03-28T15:03:25.794 INFO:tasks.workunit.client.0.smithi186.stdout: create 167/36 file after the snapshot 2024-03-28T15:03:25.796 INFO:tasks.workunit.client.0.smithi186.stdout: create 167/37 file after the snapshot 2024-03-28T15:03:25.797 INFO:tasks.workunit.client.0.smithi186.stdout: create 167/38 file after the snapshot 2024-03-28T15:03:25.799 INFO:tasks.workunit.client.0.smithi186.stdout: create 167/39 file after the snapshot 2024-03-28T15:03:25.801 INFO:tasks.workunit.client.0.smithi186.stdout: create 168/.snap/snap-subdir-test 2024-03-28T15:03:25.819 INFO:tasks.workunit.client.0.smithi186.stdout: create 168/30 file after the snapshot 2024-03-28T15:03:25.821 INFO:tasks.workunit.client.0.smithi186.stdout: create 168/31 file after the snapshot 2024-03-28T15:03:25.823 INFO:tasks.workunit.client.0.smithi186.stdout: create 168/32 file after the snapshot 2024-03-28T15:03:25.828 INFO:tasks.workunit.client.0.smithi186.stdout: create 168/33 file after the snapshot 2024-03-28T15:03:25.830 INFO:tasks.workunit.client.0.smithi186.stdout: create 168/34 file after the snapshot 2024-03-28T15:03:25.832 INFO:tasks.workunit.client.0.smithi186.stdout: create 168/35 file after the snapshot 2024-03-28T15:03:25.833 INFO:tasks.workunit.client.0.smithi186.stdout: create 168/36 file after the snapshot 2024-03-28T15:03:25.835 INFO:tasks.workunit.client.0.smithi186.stdout: create 168/37 file after the snapshot 2024-03-28T15:03:25.836 INFO:tasks.workunit.client.0.smithi186.stdout: create 168/38 file after the snapshot 2024-03-28T15:03:25.838 INFO:tasks.workunit.client.0.smithi186.stdout: create 168/39 file after the snapshot 2024-03-28T15:03:25.840 INFO:tasks.workunit.client.0.smithi186.stdout: create 169/.snap/snap-subdir-test 2024-03-28T15:03:25.858 INFO:tasks.workunit.client.0.smithi186.stdout: create 169/30 file after the snapshot 2024-03-28T15:03:25.859 INFO:tasks.workunit.client.0.smithi186.stdout: create 169/31 file after the snapshot 2024-03-28T15:03:25.860 INFO:tasks.workunit.client.0.smithi186.stdout: create 169/32 file after the snapshot 2024-03-28T15:03:25.862 INFO:tasks.workunit.client.0.smithi186.stdout: create 169/33 file after the snapshot 2024-03-28T15:03:25.864 INFO:tasks.workunit.client.0.smithi186.stdout: create 169/34 file after the snapshot 2024-03-28T15:03:25.866 INFO:tasks.workunit.client.0.smithi186.stdout: create 169/35 file after the snapshot 2024-03-28T15:03:25.868 INFO:tasks.workunit.client.0.smithi186.stdout: create 169/36 file after the snapshot 2024-03-28T15:03:25.870 INFO:tasks.workunit.client.0.smithi186.stdout: create 169/37 file after the snapshot 2024-03-28T15:03:25.871 INFO:tasks.workunit.client.0.smithi186.stdout: create 169/38 file after the snapshot 2024-03-28T15:03:25.873 INFO:tasks.workunit.client.0.smithi186.stdout: create 169/39 file after the snapshot 2024-03-28T15:03:25.875 INFO:tasks.workunit.client.0.smithi186.stdout: create 170/.snap/snap-subdir-test 2024-03-28T15:03:25.892 INFO:tasks.workunit.client.0.smithi186.stdout: create 170/30 file after the snapshot 2024-03-28T15:03:25.894 INFO:tasks.workunit.client.0.smithi186.stdout: create 170/31 file after the snapshot 2024-03-28T15:03:25.896 INFO:tasks.workunit.client.0.smithi186.stdout: create 170/32 file after the snapshot 2024-03-28T15:03:25.898 INFO:tasks.workunit.client.0.smithi186.stdout: create 170/33 file after the snapshot 2024-03-28T15:03:25.899 INFO:tasks.workunit.client.0.smithi186.stdout: create 170/34 file after the snapshot 2024-03-28T15:03:25.901 INFO:tasks.workunit.client.0.smithi186.stdout: create 170/35 file after the snapshot 2024-03-28T15:03:25.903 INFO:tasks.workunit.client.0.smithi186.stdout: create 170/36 file after the snapshot 2024-03-28T15:03:25.904 INFO:tasks.workunit.client.0.smithi186.stdout: create 170/37 file after the snapshot 2024-03-28T15:03:25.906 INFO:tasks.workunit.client.0.smithi186.stdout: create 170/38 file after the snapshot 2024-03-28T15:03:25.908 INFO:tasks.workunit.client.0.smithi186.stdout: create 170/39 file after the snapshot 2024-03-28T15:03:25.909 INFO:tasks.workunit.client.0.smithi186.stdout: create 171/.snap/snap-subdir-test 2024-03-28T15:03:25.926 INFO:tasks.workunit.client.0.smithi186.stdout: create 171/30 file after the snapshot 2024-03-28T15:03:25.931 INFO:tasks.workunit.client.0.smithi186.stdout: create 171/31 file after the snapshot 2024-03-28T15:03:25.931 INFO:tasks.workunit.client.0.smithi186.stdout: create 171/32 file after the snapshot 2024-03-28T15:03:25.931 INFO:tasks.workunit.client.0.smithi186.stdout: create 171/33 file after the snapshot 2024-03-28T15:03:25.932 INFO:tasks.workunit.client.0.smithi186.stdout: create 171/34 file after the snapshot 2024-03-28T15:03:25.934 INFO:tasks.workunit.client.0.smithi186.stdout: create 171/35 file after the snapshot 2024-03-28T15:03:25.935 INFO:tasks.workunit.client.0.smithi186.stdout: create 171/36 file after the snapshot 2024-03-28T15:03:25.937 INFO:tasks.workunit.client.0.smithi186.stdout: create 171/37 file after the snapshot 2024-03-28T15:03:25.939 INFO:tasks.workunit.client.0.smithi186.stdout: create 171/38 file after the snapshot 2024-03-28T15:03:25.940 INFO:tasks.workunit.client.0.smithi186.stdout: create 171/39 file after the snapshot 2024-03-28T15:03:25.942 INFO:tasks.workunit.client.0.smithi186.stdout: create 172/.snap/snap-subdir-test 2024-03-28T15:03:25.959 INFO:tasks.workunit.client.0.smithi186.stdout: create 172/30 file after the snapshot 2024-03-28T15:03:25.961 INFO:tasks.workunit.client.0.smithi186.stdout: create 172/31 file after the snapshot 2024-03-28T15:03:25.963 INFO:tasks.workunit.client.0.smithi186.stdout: create 172/32 file after the snapshot 2024-03-28T15:03:25.964 INFO:tasks.workunit.client.0.smithi186.stdout: create 172/33 file after the snapshot 2024-03-28T15:03:25.966 INFO:tasks.workunit.client.0.smithi186.stdout: create 172/34 file after the snapshot 2024-03-28T15:03:25.968 INFO:tasks.workunit.client.0.smithi186.stdout: create 172/35 file after the snapshot 2024-03-28T15:03:25.970 INFO:tasks.workunit.client.0.smithi186.stdout: create 172/36 file after the snapshot 2024-03-28T15:03:25.972 INFO:tasks.workunit.client.0.smithi186.stdout: create 172/37 file after the snapshot 2024-03-28T15:03:25.974 INFO:tasks.workunit.client.0.smithi186.stdout: create 172/38 file after the snapshot 2024-03-28T15:03:25.975 INFO:tasks.workunit.client.0.smithi186.stdout: create 172/39 file after the snapshot 2024-03-28T15:03:25.977 INFO:tasks.workunit.client.0.smithi186.stdout: create 173/.snap/snap-subdir-test 2024-03-28T15:03:25.995 INFO:tasks.workunit.client.0.smithi186.stdout: create 173/30 file after the snapshot 2024-03-28T15:03:25.997 INFO:tasks.workunit.client.0.smithi186.stdout: create 173/31 file after the snapshot 2024-03-28T15:03:25.998 INFO:tasks.workunit.client.0.smithi186.stdout: create 173/32 file after the snapshot 2024-03-28T15:03:26.000 INFO:tasks.workunit.client.0.smithi186.stdout: create 173/33 file after the snapshot 2024-03-28T15:03:26.002 INFO:tasks.workunit.client.0.smithi186.stdout: create 173/34 file after the snapshot 2024-03-28T15:03:26.003 INFO:tasks.workunit.client.0.smithi186.stdout: create 173/35 file after the snapshot 2024-03-28T15:03:26.005 INFO:tasks.workunit.client.0.smithi186.stdout: create 173/36 file after the snapshot 2024-03-28T15:03:26.007 INFO:tasks.workunit.client.0.smithi186.stdout: create 173/37 file after the snapshot 2024-03-28T15:03:26.008 INFO:tasks.workunit.client.0.smithi186.stdout: create 173/38 file after the snapshot 2024-03-28T15:03:26.010 INFO:tasks.workunit.client.0.smithi186.stdout: create 173/39 file after the snapshot 2024-03-28T15:03:26.012 INFO:tasks.workunit.client.0.smithi186.stdout: create 174/.snap/snap-subdir-test 2024-03-28T15:03:26.029 INFO:tasks.workunit.client.0.smithi186.stdout: create 174/30 file after the snapshot 2024-03-28T15:03:26.031 INFO:tasks.workunit.client.0.smithi186.stdout: create 174/31 file after the snapshot 2024-03-28T15:03:26.032 INFO:tasks.workunit.client.0.smithi186.stdout: create 174/32 file after the snapshot 2024-03-28T15:03:26.034 INFO:tasks.workunit.client.0.smithi186.stdout: create 174/33 file after the snapshot 2024-03-28T15:03:26.036 INFO:tasks.workunit.client.0.smithi186.stdout: create 174/34 file after the snapshot 2024-03-28T15:03:26.038 INFO:tasks.workunit.client.0.smithi186.stdout: create 174/35 file after the snapshot 2024-03-28T15:03:26.040 INFO:tasks.workunit.client.0.smithi186.stdout: create 174/36 file after the snapshot 2024-03-28T15:03:26.042 INFO:tasks.workunit.client.0.smithi186.stdout: create 174/37 file after the snapshot 2024-03-28T15:03:26.044 INFO:tasks.workunit.client.0.smithi186.stdout: create 174/38 file after the snapshot 2024-03-28T15:03:26.045 INFO:tasks.workunit.client.0.smithi186.stdout: create 174/39 file after the snapshot 2024-03-28T15:03:26.047 INFO:tasks.workunit.client.0.smithi186.stdout: create 175/.snap/snap-subdir-test 2024-03-28T15:03:26.065 INFO:tasks.workunit.client.0.smithi186.stdout: create 175/30 file after the snapshot 2024-03-28T15:03:26.067 INFO:tasks.workunit.client.0.smithi186.stdout: create 175/31 file after the snapshot 2024-03-28T15:03:26.069 INFO:tasks.workunit.client.0.smithi186.stdout: create 175/32 file after the snapshot 2024-03-28T15:03:26.070 INFO:tasks.workunit.client.0.smithi186.stdout: create 175/33 file after the snapshot 2024-03-28T15:03:26.072 INFO:tasks.workunit.client.0.smithi186.stdout: create 175/34 file after the snapshot 2024-03-28T15:03:26.074 INFO:tasks.workunit.client.0.smithi186.stdout: create 175/35 file after the snapshot 2024-03-28T15:03:26.076 INFO:tasks.workunit.client.0.smithi186.stdout: create 175/36 file after the snapshot 2024-03-28T15:03:26.078 INFO:tasks.workunit.client.0.smithi186.stdout: create 175/37 file after the snapshot 2024-03-28T15:03:26.080 INFO:tasks.workunit.client.0.smithi186.stdout: create 175/38 file after the snapshot 2024-03-28T15:03:26.082 INFO:tasks.workunit.client.0.smithi186.stdout: create 175/39 file after the snapshot 2024-03-28T15:03:26.083 INFO:tasks.workunit.client.0.smithi186.stdout: create 176/.snap/snap-subdir-test 2024-03-28T15:03:26.100 INFO:tasks.workunit.client.0.smithi186.stdout: create 176/30 file after the snapshot 2024-03-28T15:03:26.101 INFO:tasks.workunit.client.0.smithi186.stdout: create 176/31 file after the snapshot 2024-03-28T15:03:26.103 INFO:tasks.workunit.client.0.smithi186.stdout: create 176/32 file after the snapshot 2024-03-28T15:03:26.105 INFO:tasks.workunit.client.0.smithi186.stdout: create 176/33 file after the snapshot 2024-03-28T15:03:26.107 INFO:tasks.workunit.client.0.smithi186.stdout: create 176/34 file after the snapshot 2024-03-28T15:03:26.108 INFO:tasks.workunit.client.0.smithi186.stdout: create 176/35 file after the snapshot 2024-03-28T15:03:26.110 INFO:tasks.workunit.client.0.smithi186.stdout: create 176/36 file after the snapshot 2024-03-28T15:03:26.112 INFO:tasks.workunit.client.0.smithi186.stdout: create 176/37 file after the snapshot 2024-03-28T15:03:26.114 INFO:tasks.workunit.client.0.smithi186.stdout: create 176/38 file after the snapshot 2024-03-28T15:03:26.116 INFO:tasks.workunit.client.0.smithi186.stdout: create 176/39 file after the snapshot 2024-03-28T15:03:26.118 INFO:tasks.workunit.client.0.smithi186.stdout: create 177/.snap/snap-subdir-test 2024-03-28T15:03:26.136 INFO:tasks.workunit.client.0.smithi186.stdout: create 177/30 file after the snapshot 2024-03-28T15:03:26.137 INFO:tasks.workunit.client.0.smithi186.stdout: create 177/31 file after the snapshot 2024-03-28T15:03:26.139 INFO:tasks.workunit.client.0.smithi186.stdout: create 177/32 file after the snapshot 2024-03-28T15:03:26.141 INFO:tasks.workunit.client.0.smithi186.stdout: create 177/33 file after the snapshot 2024-03-28T15:03:26.143 INFO:tasks.workunit.client.0.smithi186.stdout: create 177/34 file after the snapshot 2024-03-28T15:03:26.144 INFO:tasks.workunit.client.0.smithi186.stdout: create 177/35 file after the snapshot 2024-03-28T15:03:26.146 INFO:tasks.workunit.client.0.smithi186.stdout: create 177/36 file after the snapshot 2024-03-28T15:03:26.147 INFO:tasks.workunit.client.0.smithi186.stdout: create 177/37 file after the snapshot 2024-03-28T15:03:26.149 INFO:tasks.workunit.client.0.smithi186.stdout: create 177/38 file after the snapshot 2024-03-28T15:03:26.151 INFO:tasks.workunit.client.0.smithi186.stdout: create 177/39 file after the snapshot 2024-03-28T15:03:26.153 INFO:tasks.workunit.client.0.smithi186.stdout: create 178/.snap/snap-subdir-test 2024-03-28T15:03:26.169 INFO:tasks.workunit.client.0.smithi186.stdout: create 178/30 file after the snapshot 2024-03-28T15:03:26.171 INFO:tasks.workunit.client.0.smithi186.stdout: create 178/31 file after the snapshot 2024-03-28T15:03:26.173 INFO:tasks.workunit.client.0.smithi186.stdout: create 178/32 file after the snapshot 2024-03-28T15:03:26.174 INFO:tasks.workunit.client.0.smithi186.stdout: create 178/33 file after the snapshot 2024-03-28T15:03:26.176 INFO:tasks.workunit.client.0.smithi186.stdout: create 178/34 file after the snapshot 2024-03-28T15:03:26.177 INFO:tasks.workunit.client.0.smithi186.stdout: create 178/35 file after the snapshot 2024-03-28T15:03:26.180 INFO:tasks.workunit.client.0.smithi186.stdout: create 178/36 file after the snapshot 2024-03-28T15:03:26.181 INFO:tasks.workunit.client.0.smithi186.stdout: create 178/37 file after the snapshot 2024-03-28T15:03:26.184 INFO:tasks.workunit.client.0.smithi186.stdout: create 178/38 file after the snapshot 2024-03-28T15:03:26.187 INFO:tasks.workunit.client.0.smithi186.stdout: create 178/39 file after the snapshot 2024-03-28T15:03:26.189 INFO:tasks.workunit.client.0.smithi186.stdout: create 179/.snap/snap-subdir-test 2024-03-28T15:03:26.216 INFO:tasks.workunit.client.0.smithi186.stdout: create 179/30 file after the snapshot 2024-03-28T15:03:26.218 INFO:tasks.workunit.client.0.smithi186.stdout: create 179/31 file after the snapshot 2024-03-28T15:03:26.220 INFO:tasks.workunit.client.0.smithi186.stdout: create 179/32 file after the snapshot 2024-03-28T15:03:26.222 INFO:tasks.workunit.client.0.smithi186.stdout: create 179/33 file after the snapshot 2024-03-28T15:03:26.224 INFO:tasks.workunit.client.0.smithi186.stdout: create 179/34 file after the snapshot 2024-03-28T15:03:26.225 INFO:tasks.workunit.client.0.smithi186.stdout: create 179/35 file after the snapshot 2024-03-28T15:03:26.227 INFO:tasks.workunit.client.0.smithi186.stdout: create 179/36 file after the snapshot 2024-03-28T15:03:26.229 INFO:tasks.workunit.client.0.smithi186.stdout: create 179/37 file after the snapshot 2024-03-28T15:03:26.231 INFO:tasks.workunit.client.0.smithi186.stdout: create 179/38 file after the snapshot 2024-03-28T15:03:26.233 INFO:tasks.workunit.client.0.smithi186.stdout: create 179/39 file after the snapshot 2024-03-28T15:03:26.234 INFO:tasks.workunit.client.0.smithi186.stdout: create 180/.snap/snap-subdir-test 2024-03-28T15:03:26.253 INFO:tasks.workunit.client.0.smithi186.stdout: create 180/30 file after the snapshot 2024-03-28T15:03:26.254 INFO:tasks.workunit.client.0.smithi186.stdout: create 180/31 file after the snapshot 2024-03-28T15:03:26.256 INFO:tasks.workunit.client.0.smithi186.stdout: create 180/32 file after the snapshot 2024-03-28T15:03:26.258 INFO:tasks.workunit.client.0.smithi186.stdout: create 180/33 file after the snapshot 2024-03-28T15:03:26.260 INFO:tasks.workunit.client.0.smithi186.stdout: create 180/34 file after the snapshot 2024-03-28T15:03:26.262 INFO:tasks.workunit.client.0.smithi186.stdout: create 180/35 file after the snapshot 2024-03-28T15:03:26.264 INFO:tasks.workunit.client.0.smithi186.stdout: create 180/36 file after the snapshot 2024-03-28T15:03:26.265 INFO:tasks.workunit.client.0.smithi186.stdout: create 180/37 file after the snapshot 2024-03-28T15:03:26.267 INFO:tasks.workunit.client.0.smithi186.stdout: create 180/38 file after the snapshot 2024-03-28T15:03:26.270 INFO:tasks.workunit.client.0.smithi186.stdout: create 180/39 file after the snapshot 2024-03-28T15:03:26.272 INFO:tasks.workunit.client.0.smithi186.stdout: create 181/.snap/snap-subdir-test 2024-03-28T15:03:26.286 INFO:tasks.workunit.client.0.smithi186.stdout: create 181/30 file after the snapshot 2024-03-28T15:03:26.288 INFO:tasks.workunit.client.0.smithi186.stdout: create 181/31 file after the snapshot 2024-03-28T15:03:26.290 INFO:tasks.workunit.client.0.smithi186.stdout: create 181/32 file after the snapshot 2024-03-28T15:03:26.291 INFO:tasks.workunit.client.0.smithi186.stdout: create 181/33 file after the snapshot 2024-03-28T15:03:26.293 INFO:tasks.workunit.client.0.smithi186.stdout: create 181/34 file after the snapshot 2024-03-28T15:03:26.295 INFO:tasks.workunit.client.0.smithi186.stdout: create 181/35 file after the snapshot 2024-03-28T15:03:26.297 INFO:tasks.workunit.client.0.smithi186.stdout: create 181/36 file after the snapshot 2024-03-28T15:03:26.298 INFO:tasks.workunit.client.0.smithi186.stdout: create 181/37 file after the snapshot 2024-03-28T15:03:26.300 INFO:tasks.workunit.client.0.smithi186.stdout: create 181/38 file after the snapshot 2024-03-28T15:03:26.302 INFO:tasks.workunit.client.0.smithi186.stdout: create 181/39 file after the snapshot 2024-03-28T15:03:26.303 INFO:tasks.workunit.client.0.smithi186.stdout: create 182/.snap/snap-subdir-test 2024-03-28T15:03:26.326 INFO:tasks.workunit.client.0.smithi186.stdout: create 182/30 file after the snapshot 2024-03-28T15:03:26.327 INFO:tasks.workunit.client.0.smithi186.stdout: create 182/31 file after the snapshot 2024-03-28T15:03:26.330 INFO:tasks.workunit.client.0.smithi186.stdout: create 182/32 file after the snapshot 2024-03-28T15:03:26.331 INFO:tasks.workunit.client.0.smithi186.stdout: create 182/33 file after the snapshot 2024-03-28T15:03:26.333 INFO:tasks.workunit.client.0.smithi186.stdout: create 182/34 file after the snapshot 2024-03-28T15:03:26.335 INFO:tasks.workunit.client.0.smithi186.stdout: create 182/35 file after the snapshot 2024-03-28T15:03:26.336 INFO:tasks.workunit.client.0.smithi186.stdout: create 182/36 file after the snapshot 2024-03-28T15:03:26.338 INFO:tasks.workunit.client.0.smithi186.stdout: create 182/37 file after the snapshot 2024-03-28T15:03:26.340 INFO:tasks.workunit.client.0.smithi186.stdout: create 182/38 file after the snapshot 2024-03-28T15:03:26.342 INFO:tasks.workunit.client.0.smithi186.stdout: create 182/39 file after the snapshot 2024-03-28T15:03:26.344 INFO:tasks.workunit.client.0.smithi186.stdout: create 183/.snap/snap-subdir-test 2024-03-28T15:03:26.361 INFO:tasks.workunit.client.0.smithi186.stdout: create 183/30 file after the snapshot 2024-03-28T15:03:26.363 INFO:tasks.workunit.client.0.smithi186.stdout: create 183/31 file after the snapshot 2024-03-28T15:03:26.365 INFO:tasks.workunit.client.0.smithi186.stdout: create 183/32 file after the snapshot 2024-03-28T15:03:26.370 INFO:tasks.workunit.client.0.smithi186.stdout: create 183/33 file after the snapshot 2024-03-28T15:03:26.371 INFO:tasks.workunit.client.0.smithi186.stdout: create 183/34 file after the snapshot 2024-03-28T15:03:26.371 INFO:tasks.workunit.client.0.smithi186.stdout: create 183/35 file after the snapshot 2024-03-28T15:03:26.371 INFO:tasks.workunit.client.0.smithi186.stdout: create 183/36 file after the snapshot 2024-03-28T15:03:26.373 INFO:tasks.workunit.client.0.smithi186.stdout: create 183/37 file after the snapshot 2024-03-28T15:03:26.374 INFO:tasks.workunit.client.0.smithi186.stdout: create 183/38 file after the snapshot 2024-03-28T15:03:26.376 INFO:tasks.workunit.client.0.smithi186.stdout: create 183/39 file after the snapshot 2024-03-28T15:03:26.378 INFO:tasks.workunit.client.0.smithi186.stdout: create 184/.snap/snap-subdir-test 2024-03-28T15:03:26.395 INFO:tasks.workunit.client.0.smithi186.stdout: create 184/30 file after the snapshot 2024-03-28T15:03:26.396 INFO:tasks.workunit.client.0.smithi186.stdout: create 184/31 file after the snapshot 2024-03-28T15:03:26.398 INFO:tasks.workunit.client.0.smithi186.stdout: create 184/32 file after the snapshot 2024-03-28T15:03:26.400 INFO:tasks.workunit.client.0.smithi186.stdout: create 184/33 file after the snapshot 2024-03-28T15:03:26.401 INFO:tasks.workunit.client.0.smithi186.stdout: create 184/34 file after the snapshot 2024-03-28T15:03:26.403 INFO:tasks.workunit.client.0.smithi186.stdout: create 184/35 file after the snapshot 2024-03-28T15:03:26.405 INFO:tasks.workunit.client.0.smithi186.stdout: create 184/36 file after the snapshot 2024-03-28T15:03:26.407 INFO:tasks.workunit.client.0.smithi186.stdout: create 184/37 file after the snapshot 2024-03-28T15:03:26.409 INFO:tasks.workunit.client.0.smithi186.stdout: create 184/38 file after the snapshot 2024-03-28T15:03:26.411 INFO:tasks.workunit.client.0.smithi186.stdout: create 184/39 file after the snapshot 2024-03-28T15:03:26.413 INFO:tasks.workunit.client.0.smithi186.stdout: create 185/.snap/snap-subdir-test 2024-03-28T15:03:26.431 INFO:tasks.workunit.client.0.smithi186.stdout: create 185/30 file after the snapshot 2024-03-28T15:03:26.433 INFO:tasks.workunit.client.0.smithi186.stdout: create 185/31 file after the snapshot 2024-03-28T15:03:26.435 INFO:tasks.workunit.client.0.smithi186.stdout: create 185/32 file after the snapshot 2024-03-28T15:03:26.437 INFO:tasks.workunit.client.0.smithi186.stdout: create 185/33 file after the snapshot 2024-03-28T15:03:26.438 INFO:tasks.workunit.client.0.smithi186.stdout: create 185/34 file after the snapshot 2024-03-28T15:03:26.439 INFO:tasks.workunit.client.0.smithi186.stdout: create 185/35 file after the snapshot 2024-03-28T15:03:26.441 INFO:tasks.workunit.client.0.smithi186.stdout: create 185/36 file after the snapshot 2024-03-28T15:03:26.443 INFO:tasks.workunit.client.0.smithi186.stdout: create 185/37 file after the snapshot 2024-03-28T15:03:26.445 INFO:tasks.workunit.client.0.smithi186.stdout: create 185/38 file after the snapshot 2024-03-28T15:03:26.447 INFO:tasks.workunit.client.0.smithi186.stdout: create 185/39 file after the snapshot 2024-03-28T15:03:26.449 INFO:tasks.workunit.client.0.smithi186.stdout: create 186/.snap/snap-subdir-test 2024-03-28T15:03:26.466 INFO:tasks.workunit.client.0.smithi186.stdout: create 186/30 file after the snapshot 2024-03-28T15:03:26.468 INFO:tasks.workunit.client.0.smithi186.stdout: create 186/31 file after the snapshot 2024-03-28T15:03:26.470 INFO:tasks.workunit.client.0.smithi186.stdout: create 186/32 file after the snapshot 2024-03-28T15:03:26.472 INFO:tasks.workunit.client.0.smithi186.stdout: create 186/33 file after the snapshot 2024-03-28T15:03:26.474 INFO:tasks.workunit.client.0.smithi186.stdout: create 186/34 file after the snapshot 2024-03-28T15:03:26.475 INFO:tasks.workunit.client.0.smithi186.stdout: create 186/35 file after the snapshot 2024-03-28T15:03:26.477 INFO:tasks.workunit.client.0.smithi186.stdout: create 186/36 file after the snapshot 2024-03-28T15:03:26.479 INFO:tasks.workunit.client.0.smithi186.stdout: create 186/37 file after the snapshot 2024-03-28T15:03:26.481 INFO:tasks.workunit.client.0.smithi186.stdout: create 186/38 file after the snapshot 2024-03-28T15:03:26.482 INFO:tasks.workunit.client.0.smithi186.stdout: create 186/39 file after the snapshot 2024-03-28T15:03:26.484 INFO:tasks.workunit.client.0.smithi186.stdout: create 187/.snap/snap-subdir-test 2024-03-28T15:03:26.501 INFO:tasks.workunit.client.0.smithi186.stdout: create 187/30 file after the snapshot 2024-03-28T15:03:26.503 INFO:tasks.workunit.client.0.smithi186.stdout: create 187/31 file after the snapshot 2024-03-28T15:03:26.505 INFO:tasks.workunit.client.0.smithi186.stdout: create 187/32 file after the snapshot 2024-03-28T15:03:26.507 INFO:tasks.workunit.client.0.smithi186.stdout: create 187/33 file after the snapshot 2024-03-28T15:03:26.509 INFO:tasks.workunit.client.0.smithi186.stdout: create 187/34 file after the snapshot 2024-03-28T15:03:26.511 INFO:tasks.workunit.client.0.smithi186.stdout: create 187/35 file after the snapshot 2024-03-28T15:03:26.513 INFO:tasks.workunit.client.0.smithi186.stdout: create 187/36 file after the snapshot 2024-03-28T15:03:26.515 INFO:tasks.workunit.client.0.smithi186.stdout: create 187/37 file after the snapshot 2024-03-28T15:03:26.516 INFO:tasks.workunit.client.0.smithi186.stdout: create 187/38 file after the snapshot 2024-03-28T15:03:26.518 INFO:tasks.workunit.client.0.smithi186.stdout: create 187/39 file after the snapshot 2024-03-28T15:03:26.520 INFO:tasks.workunit.client.0.smithi186.stdout: create 188/.snap/snap-subdir-test 2024-03-28T15:03:26.538 INFO:tasks.workunit.client.0.smithi186.stdout: create 188/30 file after the snapshot 2024-03-28T15:03:26.540 INFO:tasks.workunit.client.0.smithi186.stdout: create 188/31 file after the snapshot 2024-03-28T15:03:26.542 INFO:tasks.workunit.client.0.smithi186.stdout: create 188/32 file after the snapshot 2024-03-28T15:03:26.546 INFO:tasks.workunit.client.0.smithi186.stdout: create 188/33 file after the snapshot 2024-03-28T15:03:26.546 INFO:tasks.workunit.client.0.smithi186.stdout: create 188/34 file after the snapshot 2024-03-28T15:03:26.547 INFO:tasks.workunit.client.0.smithi186.stdout: create 188/35 file after the snapshot 2024-03-28T15:03:26.548 INFO:tasks.workunit.client.0.smithi186.stdout: create 188/36 file after the snapshot 2024-03-28T15:03:26.550 INFO:tasks.workunit.client.0.smithi186.stdout: create 188/37 file after the snapshot 2024-03-28T15:03:26.552 INFO:tasks.workunit.client.0.smithi186.stdout: create 188/38 file after the snapshot 2024-03-28T15:03:26.554 INFO:tasks.workunit.client.0.smithi186.stdout: create 188/39 file after the snapshot 2024-03-28T15:03:26.556 INFO:tasks.workunit.client.0.smithi186.stdout: create 189/.snap/snap-subdir-test 2024-03-28T15:03:26.573 INFO:tasks.workunit.client.0.smithi186.stdout: create 189/30 file after the snapshot 2024-03-28T15:03:26.575 INFO:tasks.workunit.client.0.smithi186.stdout: create 189/31 file after the snapshot 2024-03-28T15:03:26.577 INFO:tasks.workunit.client.0.smithi186.stdout: create 189/32 file after the snapshot 2024-03-28T15:03:26.578 INFO:tasks.workunit.client.0.smithi186.stdout: create 189/33 file after the snapshot 2024-03-28T15:03:26.580 INFO:tasks.workunit.client.0.smithi186.stdout: create 189/34 file after the snapshot 2024-03-28T15:03:26.582 INFO:tasks.workunit.client.0.smithi186.stdout: create 189/35 file after the snapshot 2024-03-28T15:03:26.583 INFO:tasks.workunit.client.0.smithi186.stdout: create 189/36 file after the snapshot 2024-03-28T15:03:26.585 INFO:tasks.workunit.client.0.smithi186.stdout: create 189/37 file after the snapshot 2024-03-28T15:03:26.587 INFO:tasks.workunit.client.0.smithi186.stdout: create 189/38 file after the snapshot 2024-03-28T15:03:26.589 INFO:tasks.workunit.client.0.smithi186.stdout: create 189/39 file after the snapshot 2024-03-28T15:03:26.590 INFO:tasks.workunit.client.0.smithi186.stdout: create 190/.snap/snap-subdir-test 2024-03-28T15:03:26.608 INFO:tasks.workunit.client.0.smithi186.stdout: create 190/30 file after the snapshot 2024-03-28T15:03:26.610 INFO:tasks.workunit.client.0.smithi186.stdout: create 190/31 file after the snapshot 2024-03-28T15:03:26.611 INFO:tasks.workunit.client.0.smithi186.stdout: create 190/32 file after the snapshot 2024-03-28T15:03:26.613 INFO:tasks.workunit.client.0.smithi186.stdout: create 190/33 file after the snapshot 2024-03-28T15:03:26.615 INFO:tasks.workunit.client.0.smithi186.stdout: create 190/34 file after the snapshot 2024-03-28T15:03:26.616 INFO:tasks.workunit.client.0.smithi186.stdout: create 190/35 file after the snapshot 2024-03-28T15:03:26.618 INFO:tasks.workunit.client.0.smithi186.stdout: create 190/36 file after the snapshot 2024-03-28T15:03:26.620 INFO:tasks.workunit.client.0.smithi186.stdout: create 190/37 file after the snapshot 2024-03-28T15:03:26.622 INFO:tasks.workunit.client.0.smithi186.stdout: create 190/38 file after the snapshot 2024-03-28T15:03:26.624 INFO:tasks.workunit.client.0.smithi186.stdout: create 190/39 file after the snapshot 2024-03-28T15:03:26.625 INFO:tasks.workunit.client.0.smithi186.stdout: create 191/.snap/snap-subdir-test 2024-03-28T15:03:26.646 INFO:tasks.workunit.client.0.smithi186.stdout: create 191/30 file after the snapshot 2024-03-28T15:03:26.648 INFO:tasks.workunit.client.0.smithi186.stdout: create 191/31 file after the snapshot 2024-03-28T15:03:26.650 INFO:tasks.workunit.client.0.smithi186.stdout: create 191/32 file after the snapshot 2024-03-28T15:03:26.651 INFO:tasks.workunit.client.0.smithi186.stdout: create 191/33 file after the snapshot 2024-03-28T15:03:26.653 INFO:tasks.workunit.client.0.smithi186.stdout: create 191/34 file after the snapshot 2024-03-28T15:03:26.655 INFO:tasks.workunit.client.0.smithi186.stdout: create 191/35 file after the snapshot 2024-03-28T15:03:26.656 INFO:tasks.workunit.client.0.smithi186.stdout: create 191/36 file after the snapshot 2024-03-28T15:03:26.658 INFO:tasks.workunit.client.0.smithi186.stdout: create 191/37 file after the snapshot 2024-03-28T15:03:26.659 INFO:tasks.workunit.client.0.smithi186.stdout: create 191/38 file after the snapshot 2024-03-28T15:03:26.661 INFO:tasks.workunit.client.0.smithi186.stdout: create 191/39 file after the snapshot 2024-03-28T15:03:26.663 INFO:tasks.workunit.client.0.smithi186.stdout: create 192/.snap/snap-subdir-test 2024-03-28T15:03:26.680 INFO:tasks.workunit.client.0.smithi186.stdout: create 192/30 file after the snapshot 2024-03-28T15:03:26.681 INFO:tasks.workunit.client.0.smithi186.stdout: create 192/31 file after the snapshot 2024-03-28T15:03:26.683 INFO:tasks.workunit.client.0.smithi186.stdout: create 192/32 file after the snapshot 2024-03-28T15:03:26.685 INFO:tasks.workunit.client.0.smithi186.stdout: create 192/33 file after the snapshot 2024-03-28T15:03:26.686 INFO:tasks.workunit.client.0.smithi186.stdout: create 192/34 file after the snapshot 2024-03-28T15:03:26.688 INFO:tasks.workunit.client.0.smithi186.stdout: create 192/35 file after the snapshot 2024-03-28T15:03:26.690 INFO:tasks.workunit.client.0.smithi186.stdout: create 192/36 file after the snapshot 2024-03-28T15:03:26.692 INFO:tasks.workunit.client.0.smithi186.stdout: create 192/37 file after the snapshot 2024-03-28T15:03:26.693 INFO:tasks.workunit.client.0.smithi186.stdout: create 192/38 file after the snapshot 2024-03-28T15:03:26.695 INFO:tasks.workunit.client.0.smithi186.stdout: create 192/39 file after the snapshot 2024-03-28T15:03:26.697 INFO:tasks.workunit.client.0.smithi186.stdout: create 193/.snap/snap-subdir-test 2024-03-28T15:03:26.713 INFO:tasks.workunit.client.0.smithi186.stdout: create 193/30 file after the snapshot 2024-03-28T15:03:26.715 INFO:tasks.workunit.client.0.smithi186.stdout: create 193/31 file after the snapshot 2024-03-28T15:03:26.717 INFO:tasks.workunit.client.0.smithi186.stdout: create 193/32 file after the snapshot 2024-03-28T15:03:26.719 INFO:tasks.workunit.client.0.smithi186.stdout: create 193/33 file after the snapshot 2024-03-28T15:03:26.720 INFO:tasks.workunit.client.0.smithi186.stdout: create 193/34 file after the snapshot 2024-03-28T15:03:26.722 INFO:tasks.workunit.client.0.smithi186.stdout: create 193/35 file after the snapshot 2024-03-28T15:03:26.724 INFO:tasks.workunit.client.0.smithi186.stdout: create 193/36 file after the snapshot 2024-03-28T15:03:26.726 INFO:tasks.workunit.client.0.smithi186.stdout: create 193/37 file after the snapshot 2024-03-28T15:03:26.728 INFO:tasks.workunit.client.0.smithi186.stdout: create 193/38 file after the snapshot 2024-03-28T15:03:26.729 INFO:tasks.workunit.client.0.smithi186.stdout: create 193/39 file after the snapshot 2024-03-28T15:03:26.731 INFO:tasks.workunit.client.0.smithi186.stdout: create 194/.snap/snap-subdir-test 2024-03-28T15:03:26.748 INFO:tasks.workunit.client.0.smithi186.stdout: create 194/30 file after the snapshot 2024-03-28T15:03:26.749 INFO:tasks.workunit.client.0.smithi186.stdout: create 194/31 file after the snapshot 2024-03-28T15:03:26.751 INFO:tasks.workunit.client.0.smithi186.stdout: create 194/32 file after the snapshot 2024-03-28T15:03:26.753 INFO:tasks.workunit.client.0.smithi186.stdout: create 194/33 file after the snapshot 2024-03-28T15:03:26.754 INFO:tasks.workunit.client.0.smithi186.stdout: create 194/34 file after the snapshot 2024-03-28T15:03:26.756 INFO:tasks.workunit.client.0.smithi186.stdout: create 194/35 file after the snapshot 2024-03-28T15:03:26.758 INFO:tasks.workunit.client.0.smithi186.stdout: create 194/36 file after the snapshot 2024-03-28T15:03:26.759 INFO:tasks.workunit.client.0.smithi186.stdout: create 194/37 file after the snapshot 2024-03-28T15:03:26.761 INFO:tasks.workunit.client.0.smithi186.stdout: create 194/38 file after the snapshot 2024-03-28T15:03:26.763 INFO:tasks.workunit.client.0.smithi186.stdout: create 194/39 file after the snapshot 2024-03-28T15:03:26.764 INFO:tasks.workunit.client.0.smithi186.stdout: create 195/.snap/snap-subdir-test 2024-03-28T15:03:26.781 INFO:tasks.workunit.client.0.smithi186.stdout: create 195/30 file after the snapshot 2024-03-28T15:03:26.783 INFO:tasks.workunit.client.0.smithi186.stdout: create 195/31 file after the snapshot 2024-03-28T15:03:26.785 INFO:tasks.workunit.client.0.smithi186.stdout: create 195/32 file after the snapshot 2024-03-28T15:03:26.787 INFO:tasks.workunit.client.0.smithi186.stdout: create 195/33 file after the snapshot 2024-03-28T15:03:26.788 INFO:tasks.workunit.client.0.smithi186.stdout: create 195/34 file after the snapshot 2024-03-28T15:03:26.790 INFO:tasks.workunit.client.0.smithi186.stdout: create 195/35 file after the snapshot 2024-03-28T15:03:26.791 INFO:tasks.workunit.client.0.smithi186.stdout: create 195/36 file after the snapshot 2024-03-28T15:03:26.793 INFO:tasks.workunit.client.0.smithi186.stdout: create 195/37 file after the snapshot 2024-03-28T15:03:26.795 INFO:tasks.workunit.client.0.smithi186.stdout: create 195/38 file after the snapshot 2024-03-28T15:03:26.796 INFO:tasks.workunit.client.0.smithi186.stdout: create 195/39 file after the snapshot 2024-03-28T15:03:26.798 INFO:tasks.workunit.client.0.smithi186.stdout: create 196/.snap/snap-subdir-test 2024-03-28T15:03:26.813 INFO:tasks.workunit.client.0.smithi186.stdout: create 196/30 file after the snapshot 2024-03-28T15:03:26.815 INFO:tasks.workunit.client.0.smithi186.stdout: create 196/31 file after the snapshot 2024-03-28T15:03:26.817 INFO:tasks.workunit.client.0.smithi186.stdout: create 196/32 file after the snapshot 2024-03-28T15:03:26.818 INFO:tasks.workunit.client.0.smithi186.stdout: create 196/33 file after the snapshot 2024-03-28T15:03:26.820 INFO:tasks.workunit.client.0.smithi186.stdout: create 196/34 file after the snapshot 2024-03-28T15:03:26.822 INFO:tasks.workunit.client.0.smithi186.stdout: create 196/35 file after the snapshot 2024-03-28T15:03:26.824 INFO:tasks.workunit.client.0.smithi186.stdout: create 196/36 file after the snapshot 2024-03-28T15:03:26.825 INFO:tasks.workunit.client.0.smithi186.stdout: create 196/37 file after the snapshot 2024-03-28T15:03:26.827 INFO:tasks.workunit.client.0.smithi186.stdout: create 196/38 file after the snapshot 2024-03-28T15:03:26.904 INFO:tasks.workunit.client.0.smithi186.stdout: create 196/39 file after the snapshot 2024-03-28T15:03:26.906 INFO:tasks.workunit.client.0.smithi186.stdout: create 197/.snap/snap-subdir-test 2024-03-28T15:03:26.956 INFO:tasks.workunit.client.0.smithi186.stdout: create 197/30 file after the snapshot 2024-03-28T15:03:26.957 INFO:tasks.workunit.client.0.smithi186.stdout: create 197/31 file after the snapshot 2024-03-28T15:03:26.959 INFO:tasks.workunit.client.0.smithi186.stdout: create 197/32 file after the snapshot 2024-03-28T15:03:26.961 INFO:tasks.workunit.client.0.smithi186.stdout: create 197/33 file after the snapshot 2024-03-28T15:03:26.963 INFO:tasks.workunit.client.0.smithi186.stdout: create 197/34 file after the snapshot 2024-03-28T15:03:26.964 INFO:tasks.workunit.client.0.smithi186.stdout: create 197/35 file after the snapshot 2024-03-28T15:03:26.966 INFO:tasks.workunit.client.0.smithi186.stdout: create 197/36 file after the snapshot 2024-03-28T15:03:26.968 INFO:tasks.workunit.client.0.smithi186.stdout: create 197/37 file after the snapshot 2024-03-28T15:03:26.969 INFO:tasks.workunit.client.0.smithi186.stdout: create 197/38 file after the snapshot 2024-03-28T15:03:26.971 INFO:tasks.workunit.client.0.smithi186.stdout: create 197/39 file after the snapshot 2024-03-28T15:03:26.973 INFO:tasks.workunit.client.0.smithi186.stdout: create 198/.snap/snap-subdir-test 2024-03-28T15:03:26.991 INFO:tasks.workunit.client.0.smithi186.stdout: create 198/30 file after the snapshot 2024-03-28T15:03:26.993 INFO:tasks.workunit.client.0.smithi186.stdout: create 198/31 file after the snapshot 2024-03-28T15:03:26.995 INFO:tasks.workunit.client.0.smithi186.stdout: create 198/32 file after the snapshot 2024-03-28T15:03:26.997 INFO:tasks.workunit.client.0.smithi186.stdout: create 198/33 file after the snapshot 2024-03-28T15:03:26.999 INFO:tasks.workunit.client.0.smithi186.stdout: create 198/34 file after the snapshot 2024-03-28T15:03:27.001 INFO:tasks.workunit.client.0.smithi186.stdout: create 198/35 file after the snapshot 2024-03-28T15:03:27.003 INFO:tasks.workunit.client.0.smithi186.stdout: create 198/36 file after the snapshot 2024-03-28T15:03:27.004 INFO:tasks.workunit.client.0.smithi186.stdout: create 198/37 file after the snapshot 2024-03-28T15:03:27.006 INFO:tasks.workunit.client.0.smithi186.stdout: create 198/38 file after the snapshot 2024-03-28T15:03:27.007 INFO:tasks.workunit.client.0.smithi186.stdout: create 198/39 file after the snapshot 2024-03-28T15:03:27.009 INFO:tasks.workunit.client.0.smithi186.stdout: create 199/.snap/snap-subdir-test 2024-03-28T15:03:27.027 INFO:tasks.workunit.client.0.smithi186.stdout: create 199/30 file after the snapshot 2024-03-28T15:03:27.029 INFO:tasks.workunit.client.0.smithi186.stdout: create 199/31 file after the snapshot 2024-03-28T15:03:27.030 INFO:tasks.workunit.client.0.smithi186.stdout: create 199/32 file after the snapshot 2024-03-28T15:03:27.033 INFO:tasks.workunit.client.0.smithi186.stdout: create 199/33 file after the snapshot 2024-03-28T15:03:27.034 INFO:tasks.workunit.client.0.smithi186.stdout: create 199/34 file after the snapshot 2024-03-28T15:03:27.036 INFO:tasks.workunit.client.0.smithi186.stdout: create 199/35 file after the snapshot 2024-03-28T15:03:27.037 INFO:tasks.workunit.client.0.smithi186.stdout: create 199/36 file after the snapshot 2024-03-28T15:03:27.039 INFO:tasks.workunit.client.0.smithi186.stdout: create 199/37 file after the snapshot 2024-03-28T15:03:27.041 INFO:tasks.workunit.client.0.smithi186.stdout: create 199/38 file after the snapshot 2024-03-28T15:03:27.042 INFO:tasks.workunit.client.0.smithi186.stdout: create 199/39 file after the snapshot 2024-03-28T15:03:27.044 INFO:tasks.workunit.client.0.smithi186.stdout: create 200/.snap/snap-subdir-test 2024-03-28T15:03:27.065 INFO:tasks.workunit.client.0.smithi186.stdout: create 200/30 file after the snapshot 2024-03-28T15:03:27.067 INFO:tasks.workunit.client.0.smithi186.stdout: create 200/31 file after the snapshot 2024-03-28T15:03:27.069 INFO:tasks.workunit.client.0.smithi186.stdout: create 200/32 file after the snapshot 2024-03-28T15:03:27.070 INFO:tasks.workunit.client.0.smithi186.stdout: create 200/33 file after the snapshot 2024-03-28T15:03:27.072 INFO:tasks.workunit.client.0.smithi186.stdout: create 200/34 file after the snapshot 2024-03-28T15:03:27.073 INFO:tasks.workunit.client.0.smithi186.stdout: create 200/35 file after the snapshot 2024-03-28T15:03:27.075 INFO:tasks.workunit.client.0.smithi186.stdout: create 200/36 file after the snapshot 2024-03-28T15:03:27.077 INFO:tasks.workunit.client.0.smithi186.stdout: create 200/37 file after the snapshot 2024-03-28T15:03:27.078 INFO:tasks.workunit.client.0.smithi186.stdout: create 200/38 file after the snapshot 2024-03-28T15:03:27.080 INFO:tasks.workunit.client.0.smithi186.stdout: create 200/39 file after the snapshot 2024-03-28T15:03:27.082 INFO:tasks.workunit.client.0.smithi186.stdout: create 201/.snap/snap-subdir-test 2024-03-28T15:03:27.097 INFO:tasks.workunit.client.0.smithi186.stdout: create 201/30 file after the snapshot 2024-03-28T15:03:27.098 INFO:tasks.workunit.client.0.smithi186.stdout: create 201/31 file after the snapshot 2024-03-28T15:03:27.100 INFO:tasks.workunit.client.0.smithi186.stdout: create 201/32 file after the snapshot 2024-03-28T15:03:27.102 INFO:tasks.workunit.client.0.smithi186.stdout: create 201/33 file after the snapshot 2024-03-28T15:03:27.103 INFO:tasks.workunit.client.0.smithi186.stdout: create 201/34 file after the snapshot 2024-03-28T15:03:27.105 INFO:tasks.workunit.client.0.smithi186.stdout: create 201/35 file after the snapshot 2024-03-28T15:03:27.107 INFO:tasks.workunit.client.0.smithi186.stdout: create 201/36 file after the snapshot 2024-03-28T15:03:27.109 INFO:tasks.workunit.client.0.smithi186.stdout: create 201/37 file after the snapshot 2024-03-28T15:03:27.111 INFO:tasks.workunit.client.0.smithi186.stdout: create 201/38 file after the snapshot 2024-03-28T15:03:27.112 INFO:tasks.workunit.client.0.smithi186.stdout: create 201/39 file after the snapshot 2024-03-28T15:03:27.114 INFO:tasks.workunit.client.0.smithi186.stdout: create 202/.snap/snap-subdir-test 2024-03-28T15:03:27.130 INFO:tasks.workunit.client.0.smithi186.stdout: create 202/30 file after the snapshot 2024-03-28T15:03:27.132 INFO:tasks.workunit.client.0.smithi186.stdout: create 202/31 file after the snapshot 2024-03-28T15:03:27.134 INFO:tasks.workunit.client.0.smithi186.stdout: create 202/32 file after the snapshot 2024-03-28T15:03:27.135 INFO:tasks.workunit.client.0.smithi186.stdout: create 202/33 file after the snapshot 2024-03-28T15:03:27.137 INFO:tasks.workunit.client.0.smithi186.stdout: create 202/34 file after the snapshot 2024-03-28T15:03:27.138 INFO:tasks.workunit.client.0.smithi186.stdout: create 202/35 file after the snapshot 2024-03-28T15:03:27.140 INFO:tasks.workunit.client.0.smithi186.stdout: create 202/36 file after the snapshot 2024-03-28T15:03:27.141 INFO:tasks.workunit.client.0.smithi186.stdout: create 202/37 file after the snapshot 2024-03-28T15:03:27.143 INFO:tasks.workunit.client.0.smithi186.stdout: create 202/38 file after the snapshot 2024-03-28T15:03:27.145 INFO:tasks.workunit.client.0.smithi186.stdout: create 202/39 file after the snapshot 2024-03-28T15:03:27.147 INFO:tasks.workunit.client.0.smithi186.stdout: create 203/.snap/snap-subdir-test 2024-03-28T15:03:27.163 INFO:tasks.workunit.client.0.smithi186.stdout: create 203/30 file after the snapshot 2024-03-28T15:03:27.164 INFO:tasks.workunit.client.0.smithi186.stdout: create 203/31 file after the snapshot 2024-03-28T15:03:27.166 INFO:tasks.workunit.client.0.smithi186.stdout: create 203/32 file after the snapshot 2024-03-28T15:03:27.168 INFO:tasks.workunit.client.0.smithi186.stdout: create 203/33 file after the snapshot 2024-03-28T15:03:27.169 INFO:tasks.workunit.client.0.smithi186.stdout: create 203/34 file after the snapshot 2024-03-28T15:03:27.171 INFO:tasks.workunit.client.0.smithi186.stdout: create 203/35 file after the snapshot 2024-03-28T15:03:27.173 INFO:tasks.workunit.client.0.smithi186.stdout: create 203/36 file after the snapshot 2024-03-28T15:03:27.174 INFO:tasks.workunit.client.0.smithi186.stdout: create 203/37 file after the snapshot 2024-03-28T15:03:27.176 INFO:tasks.workunit.client.0.smithi186.stdout: create 203/38 file after the snapshot 2024-03-28T15:03:27.178 INFO:tasks.workunit.client.0.smithi186.stdout: create 203/39 file after the snapshot 2024-03-28T15:03:27.179 INFO:tasks.workunit.client.0.smithi186.stdout: create 204/.snap/snap-subdir-test 2024-03-28T15:03:27.195 INFO:tasks.workunit.client.0.smithi186.stdout: create 204/30 file after the snapshot 2024-03-28T15:03:27.196 INFO:tasks.workunit.client.0.smithi186.stdout: create 204/31 file after the snapshot 2024-03-28T15:03:27.198 INFO:tasks.workunit.client.0.smithi186.stdout: create 204/32 file after the snapshot 2024-03-28T15:03:27.200 INFO:tasks.workunit.client.0.smithi186.stdout: create 204/33 file after the snapshot 2024-03-28T15:03:27.201 INFO:tasks.workunit.client.0.smithi186.stdout: create 204/34 file after the snapshot 2024-03-28T15:03:27.203 INFO:tasks.workunit.client.0.smithi186.stdout: create 204/35 file after the snapshot 2024-03-28T15:03:27.205 INFO:tasks.workunit.client.0.smithi186.stdout: create 204/36 file after the snapshot 2024-03-28T15:03:27.207 INFO:tasks.workunit.client.0.smithi186.stdout: create 204/37 file after the snapshot 2024-03-28T15:03:27.209 INFO:tasks.workunit.client.0.smithi186.stdout: create 204/38 file after the snapshot 2024-03-28T15:03:27.210 INFO:tasks.workunit.client.0.smithi186.stdout: create 204/39 file after the snapshot 2024-03-28T15:03:27.212 INFO:tasks.workunit.client.0.smithi186.stdout: create 205/.snap/snap-subdir-test 2024-03-28T15:03:27.233 INFO:tasks.workunit.client.0.smithi186.stdout: create 205/30 file after the snapshot 2024-03-28T15:03:27.235 INFO:tasks.workunit.client.0.smithi186.stdout: create 205/31 file after the snapshot 2024-03-28T15:03:27.237 INFO:tasks.workunit.client.0.smithi186.stdout: create 205/32 file after the snapshot 2024-03-28T15:03:27.238 INFO:tasks.workunit.client.0.smithi186.stdout: create 205/33 file after the snapshot 2024-03-28T15:03:27.240 INFO:tasks.workunit.client.0.smithi186.stdout: create 205/34 file after the snapshot 2024-03-28T15:03:27.242 INFO:tasks.workunit.client.0.smithi186.stdout: create 205/35 file after the snapshot 2024-03-28T15:03:27.243 INFO:tasks.workunit.client.0.smithi186.stdout: create 205/36 file after the snapshot 2024-03-28T15:03:27.245 INFO:tasks.workunit.client.0.smithi186.stdout: create 205/37 file after the snapshot 2024-03-28T15:03:27.247 INFO:tasks.workunit.client.0.smithi186.stdout: create 205/38 file after the snapshot 2024-03-28T15:03:27.249 INFO:tasks.workunit.client.0.smithi186.stdout: create 205/39 file after the snapshot 2024-03-28T15:03:27.251 INFO:tasks.workunit.client.0.smithi186.stdout: create 206/.snap/snap-subdir-test 2024-03-28T15:03:27.266 INFO:tasks.workunit.client.0.smithi186.stdout: create 206/30 file after the snapshot 2024-03-28T15:03:27.267 INFO:tasks.workunit.client.0.smithi186.stdout: create 206/31 file after the snapshot 2024-03-28T15:03:27.269 INFO:tasks.workunit.client.0.smithi186.stdout: create 206/32 file after the snapshot 2024-03-28T15:03:27.271 INFO:tasks.workunit.client.0.smithi186.stdout: create 206/33 file after the snapshot 2024-03-28T15:03:27.272 INFO:tasks.workunit.client.0.smithi186.stdout: create 206/34 file after the snapshot 2024-03-28T15:03:27.274 INFO:tasks.workunit.client.0.smithi186.stdout: create 206/35 file after the snapshot 2024-03-28T15:03:27.276 INFO:tasks.workunit.client.0.smithi186.stdout: create 206/36 file after the snapshot 2024-03-28T15:03:27.278 INFO:tasks.workunit.client.0.smithi186.stdout: create 206/37 file after the snapshot 2024-03-28T15:03:27.279 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:27.280 INFO:tasks.workunit.client.0.smithi186.stdout: create 206/38 file after the snapshot 2024-03-28T15:03:27.282 INFO:tasks.workunit.client.0.smithi186.stdout: create 206/39 file after the snapshot 2024-03-28T15:03:27.283 INFO:tasks.workunit.client.0.smithi186.stdout: create 207/.snap/snap-subdir-test 2024-03-28T15:03:27.299 INFO:tasks.workunit.client.0.smithi186.stdout: create 207/30 file after the snapshot 2024-03-28T15:03:27.300 INFO:tasks.workunit.client.0.smithi186.stdout: create 207/31 file after the snapshot 2024-03-28T15:03:27.302 INFO:tasks.workunit.client.0.smithi186.stdout: create 207/32 file after the snapshot 2024-03-28T15:03:27.304 INFO:tasks.workunit.client.0.smithi186.stdout: create 207/33 file after the snapshot 2024-03-28T15:03:27.306 INFO:tasks.workunit.client.0.smithi186.stdout: create 207/34 file after the snapshot 2024-03-28T15:03:27.307 INFO:tasks.workunit.client.0.smithi186.stdout: create 207/35 file after the snapshot 2024-03-28T15:03:27.309 INFO:tasks.workunit.client.0.smithi186.stdout: create 207/36 file after the snapshot 2024-03-28T15:03:27.311 INFO:tasks.workunit.client.0.smithi186.stdout: create 207/37 file after the snapshot 2024-03-28T15:03:27.313 INFO:tasks.workunit.client.0.smithi186.stdout: create 207/38 file after the snapshot 2024-03-28T15:03:27.315 INFO:tasks.workunit.client.0.smithi186.stdout: create 207/39 file after the snapshot 2024-03-28T15:03:27.317 INFO:tasks.workunit.client.0.smithi186.stdout: create 208/.snap/snap-subdir-test 2024-03-28T15:03:27.335 INFO:tasks.workunit.client.0.smithi186.stdout: create 208/30 file after the snapshot 2024-03-28T15:03:27.336 INFO:tasks.workunit.client.0.smithi186.stdout: create 208/31 file after the snapshot 2024-03-28T15:03:27.338 INFO:tasks.workunit.client.0.smithi186.stdout: create 208/32 file after the snapshot 2024-03-28T15:03:27.340 INFO:tasks.workunit.client.0.smithi186.stdout: create 208/33 file after the snapshot 2024-03-28T15:03:27.342 INFO:tasks.workunit.client.0.smithi186.stdout: create 208/34 file after the snapshot 2024-03-28T15:03:27.343 INFO:tasks.workunit.client.0.smithi186.stdout: create 208/35 file after the snapshot 2024-03-28T15:03:27.345 INFO:tasks.workunit.client.0.smithi186.stdout: create 208/36 file after the snapshot 2024-03-28T15:03:27.347 INFO:tasks.workunit.client.0.smithi186.stdout: create 208/37 file after the snapshot 2024-03-28T15:03:27.350 INFO:tasks.workunit.client.0.smithi186.stdout: create 208/38 file after the snapshot 2024-03-28T15:03:27.351 INFO:tasks.workunit.client.0.smithi186.stdout: create 208/39 file after the snapshot 2024-03-28T15:03:27.354 INFO:tasks.workunit.client.0.smithi186.stdout: create 209/.snap/snap-subdir-test 2024-03-28T15:03:27.373 INFO:tasks.workunit.client.0.smithi186.stdout: create 209/30 file after the snapshot 2024-03-28T15:03:27.374 INFO:tasks.workunit.client.0.smithi186.stdout: create 209/31 file after the snapshot 2024-03-28T15:03:27.376 INFO:tasks.workunit.client.0.smithi186.stdout: create 209/32 file after the snapshot 2024-03-28T15:03:27.377 INFO:tasks.workunit.client.0.smithi186.stdout: create 209/33 file after the snapshot 2024-03-28T15:03:27.379 INFO:tasks.workunit.client.0.smithi186.stdout: create 209/34 file after the snapshot 2024-03-28T15:03:27.381 INFO:tasks.workunit.client.0.smithi186.stdout: create 209/35 file after the snapshot 2024-03-28T15:03:27.383 INFO:tasks.workunit.client.0.smithi186.stdout: create 209/36 file after the snapshot 2024-03-28T15:03:27.384 INFO:tasks.workunit.client.0.smithi186.stdout: create 209/37 file after the snapshot 2024-03-28T15:03:27.386 INFO:tasks.workunit.client.0.smithi186.stdout: create 209/38 file after the snapshot 2024-03-28T15:03:27.388 INFO:tasks.workunit.client.0.smithi186.stdout: create 209/39 file after the snapshot 2024-03-28T15:03:27.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.388+0000 7fd56cac9640 1 Processor -- start 2024-03-28T15:03:27.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.388+0000 7fd56cac9640 1 -- start start 2024-03-28T15:03:27.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd56cac9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5681571a0 0x7fd5680c3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd56cac9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 0x7fd5680c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd56cac9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd56806eaf0 con 0x7fd56806f410 2024-03-28T15:03:27.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd56cac9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fd5680c9070 con 0x7fd5681571a0 2024-03-28T15:03:27.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd56cac9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd5680c84f0 con 0x7fd568072410 2024-03-28T15:03:27.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd566d9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 0x7fd5680c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd56659c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5681571a0 0x7fd5680c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd566d9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 0x7fd5680c2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:55796/0 (socket says 172.21.15.186:55796) 2024-03-28T15:03:27.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd566d9d640 1 -- 172.21.15.186:0/2966050911 learned_addr learned my addr 172.21.15.186:0/2966050911 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:27.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd55ffff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd56806f410 0x7fd56815b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:49108/0 (socket says 172.21.15.186:49108) 2024-03-28T15:03:27.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd56659c640 1 -- 172.21.15.186:0/2966050911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 msgr2=0x7fd5680c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:27.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd56659c640 1 --2- 172.21.15.186:0/2966050911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 0x7fd5680c2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd56659c640 1 -- 172.21.15.186:0/2966050911 >> v1:172.21.15.191:6789/0 conn(0x7fd56806f410 legacy=0x7fd56815b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:27.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd56659c640 1 -- 172.21.15.186:0/2966050911 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5680c87d0 con 0x7fd5681571a0 2024-03-28T15:03:27.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.389+0000 7fd566d9d640 1 --2- 172.21.15.186:0/2966050911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 0x7fd5680c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:03:27.391 INFO:tasks.workunit.client.0.smithi186.stdout: create 210/.snap/snap-subdir-test 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.390+0000 7fd56659c640 1 --2- 172.21.15.186:0/2966050911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5681571a0 0x7fd5680c3790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd5580097a0 tx=0x7fd55802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a35323df4166302e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.390+0000 7fd565d9b640 1 -- 172.21.15.186:0/2966050911 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558034070 con 0x7fd5681571a0 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.390+0000 7fd565d9b640 1 -- 172.21.15.186:0/2966050911 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd558030610 con 0x7fd5681571a0 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.390+0000 7fd565d9b640 1 -- 172.21.15.186:0/2966050911 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5580309e0 con 0x7fd5681571a0 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.391+0000 7fd56cac9640 1 -- 172.21.15.186:0/2966050911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5681571a0 msgr2=0x7fd5680c3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.391+0000 7fd56cac9640 1 --2- 172.21.15.186:0/2966050911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5681571a0 0x7fd5680c3790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd5580097a0 tx=0x7fd55802b960 comp rx=0 tx=0).stop 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.391+0000 7fd56cac9640 1 -- 172.21.15.186:0/2966050911 shutdown_connections 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.391+0000 7fd56cac9640 1 --2- 172.21.15.186:0/2966050911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 0x7fd5680c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.391+0000 7fd56cac9640 1 --2- 172.21.15.186:0/2966050911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5681571a0 0x7fd5680c3790 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.391+0000 7fd56cac9640 1 -- 172.21.15.186:0/2966050911 >> 172.21.15.186:0/2966050911 conn(0x7fd5680cd360 msgr2=0x7fd5680cd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.391+0000 7fd56cac9640 1 -- 172.21.15.186:0/2966050911 shutdown_connections 2024-03-28T15:03:27.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.391+0000 7fd56cac9640 1 -- 172.21.15.186:0/2966050911 wait complete. 2024-03-28T15:03:27.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.393+0000 7fd56cac9640 1 Processor -- start 2024-03-28T15:03:27.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.393+0000 7fd56cac9640 1 -- start start 2024-03-28T15:03:27.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.393+0000 7fd56cac9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd56806f410 0x7fd56814e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.393+0000 7fd56cac9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 0x7fd56814fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.393+0000 7fd56cac9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5681571a0 0x7fd5681500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.393+0000 7fd56cac9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd56815ad90 con 0x7fd5681571a0 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.393+0000 7fd56cac9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd5680c8d90 con 0x7fd56806f410 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.393+0000 7fd56cac9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd56815b9c0 con 0x7fd568072410 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd566d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5681571a0 0x7fd5681500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd56659c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd56806f410 0x7fd56814e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd566d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5681571a0 0x7fd5681500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:35296/0 (socket says 172.21.15.186:35296) 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd55ffff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 0x7fd56814fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd56659c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd56806f410 0x7fd56814e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:38358/0 (socket says 172.21.15.186:38358) 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd566d9d640 1 -- 172.21.15.186:0/2069722342 learned_addr learned my addr 172.21.15.186:0/2069722342 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd566d9d640 1 -- 172.21.15.186:0/2069722342 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 msgr2=0x7fd56814fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd566d9d640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 0x7fd56814fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd566d9d640 1 -- 172.21.15.186:0/2069722342 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd56806f410 msgr2=0x7fd56814e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd566d9d640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd56806f410 0x7fd56814e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd566d9d640 1 -- 172.21.15.186:0/2069722342 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5680ccca0 con 0x7fd5681571a0 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd55ffff640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 0x7fd56814fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:03:27.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd566d9d640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5681571a0 0x7fd5681500d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd56000e140 tx=0x7fd56000e600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:27.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd55effd640 1 -- 172.21.15.186:0/2069722342 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56000b2d0 con 0x7fd5681571a0 2024-03-28T15:03:27.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd55effd640 1 -- 172.21.15.186:0/2069722342 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd56001a070 con 0x7fd5681571a0 2024-03-28T15:03:27.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd56cac9640 1 -- 172.21.15.186:0/2069722342 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd56814ee40 con 0x7fd5681571a0 2024-03-28T15:03:27.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd55effd640 1 -- 172.21.15.186:0/2069722342 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560009840 con 0x7fd5681571a0 2024-03-28T15:03:27.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.394+0000 7fd56cac9640 1 -- 172.21.15.186:0/2069722342 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd56814f0e0 con 0x7fd5681571a0 2024-03-28T15:03:27.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.395+0000 7fd56cac9640 1 -- 172.21.15.186:0/2069722342 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5680686d0 con 0x7fd5681571a0 2024-03-28T15:03:27.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.396+0000 7fd55effd640 1 -- 172.21.15.186:0/2069722342 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd560005d10 con 0x7fd5681571a0 2024-03-28T15:03:27.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.396+0000 7fd55effd640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd54806c560 0x7fd54806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.396+0000 7fd55effd640 1 -- 172.21.15.186:0/2069722342 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5939+0+0 (secure 0 0 0) 0x7fd56008dab0 con 0x7fd5681571a0 2024-03-28T15:03:27.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.396+0000 7fd56659c640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd54806c560 0x7fd54806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.397+0000 7fd56659c640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd54806c560 0x7fd54806ea20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd558036000 tx=0x7fd55803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:27.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.401+0000 7fd55effd640 1 -- 172.21.15.186:0/2069722342 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd5680ccca0 con 0x7fd5681571a0 2024-03-28T15:03:27.406 INFO:tasks.workunit.client.0.smithi186.stdout: create 210/30 file after the snapshot 2024-03-28T15:03:27.408 INFO:tasks.workunit.client.0.smithi186.stdout: create 210/31 file after the snapshot 2024-03-28T15:03:27.410 INFO:tasks.workunit.client.0.smithi186.stdout: create 210/32 file after the snapshot 2024-03-28T15:03:27.412 INFO:tasks.workunit.client.0.smithi186.stdout: create 210/33 file after the snapshot 2024-03-28T15:03:27.414 INFO:tasks.workunit.client.0.smithi186.stdout: create 210/34 file after the snapshot 2024-03-28T15:03:27.416 INFO:tasks.workunit.client.0.smithi186.stdout: create 210/35 file after the snapshot 2024-03-28T15:03:27.417 INFO:tasks.workunit.client.0.smithi186.stdout: create 210/36 file after the snapshot 2024-03-28T15:03:27.419 INFO:tasks.workunit.client.0.smithi186.stdout: create 210/37 file after the snapshot 2024-03-28T15:03:27.422 INFO:tasks.workunit.client.0.smithi186.stdout: create 210/38 file after the snapshot 2024-03-28T15:03:27.424 INFO:tasks.workunit.client.0.smithi186.stdout: create 210/39 file after the snapshot 2024-03-28T15:03:27.426 INFO:tasks.workunit.client.0.smithi186.stdout: create 211/.snap/snap-subdir-test 2024-03-28T15:03:27.442 INFO:tasks.workunit.client.0.smithi186.stdout: create 211/30 file after the snapshot 2024-03-28T15:03:27.444 INFO:tasks.workunit.client.0.smithi186.stdout: create 211/31 file after the snapshot 2024-03-28T15:03:27.446 INFO:tasks.workunit.client.0.smithi186.stdout: create 211/32 file after the snapshot 2024-03-28T15:03:27.447 INFO:tasks.workunit.client.0.smithi186.stdout: create 211/33 file after the snapshot 2024-03-28T15:03:27.449 INFO:tasks.workunit.client.0.smithi186.stdout: create 211/34 file after the snapshot 2024-03-28T15:03:27.451 INFO:tasks.workunit.client.0.smithi186.stdout: create 211/35 file after the snapshot 2024-03-28T15:03:27.452 INFO:tasks.workunit.client.0.smithi186.stdout: create 211/36 file after the snapshot 2024-03-28T15:03:27.454 INFO:tasks.workunit.client.0.smithi186.stdout: create 211/37 file after the snapshot 2024-03-28T15:03:27.456 INFO:tasks.workunit.client.0.smithi186.stdout: create 211/38 file after the snapshot 2024-03-28T15:03:27.457 INFO:tasks.workunit.client.0.smithi186.stdout: create 211/39 file after the snapshot 2024-03-28T15:03:27.459 INFO:tasks.workunit.client.0.smithi186.stdout: create 212/.snap/snap-subdir-test 2024-03-28T15:03:27.475 INFO:tasks.workunit.client.0.smithi186.stdout: create 212/30 file after the snapshot 2024-03-28T15:03:27.477 INFO:tasks.workunit.client.0.smithi186.stdout: create 212/31 file after the snapshot 2024-03-28T15:03:27.479 INFO:tasks.workunit.client.0.smithi186.stdout: create 212/32 file after the snapshot 2024-03-28T15:03:27.480 INFO:tasks.workunit.client.0.smithi186.stdout: create 212/33 file after the snapshot 2024-03-28T15:03:27.482 INFO:tasks.workunit.client.0.smithi186.stdout: create 212/34 file after the snapshot 2024-03-28T15:03:27.484 INFO:tasks.workunit.client.0.smithi186.stdout: create 212/35 file after the snapshot 2024-03-28T15:03:27.486 INFO:tasks.workunit.client.0.smithi186.stdout: create 212/36 file after the snapshot 2024-03-28T15:03:27.488 INFO:tasks.workunit.client.0.smithi186.stdout: create 212/37 file after the snapshot 2024-03-28T15:03:27.489 INFO:tasks.workunit.client.0.smithi186.stdout: create 212/38 file after the snapshot 2024-03-28T15:03:27.491 INFO:tasks.workunit.client.0.smithi186.stdout: create 212/39 file after the snapshot 2024-03-28T15:03:27.493 INFO:tasks.workunit.client.0.smithi186.stdout: create 213/.snap/snap-subdir-test 2024-03-28T15:03:27.513 INFO:tasks.workunit.client.0.smithi186.stdout: create 213/30 file after the snapshot 2024-03-28T15:03:27.515 INFO:tasks.workunit.client.0.smithi186.stdout: create 213/31 file after the snapshot 2024-03-28T15:03:27.517 INFO:tasks.workunit.client.0.smithi186.stdout: create 213/32 file after the snapshot 2024-03-28T15:03:27.518 INFO:tasks.workunit.client.0.smithi186.stdout: create 213/33 file after the snapshot 2024-03-28T15:03:27.520 INFO:tasks.workunit.client.0.smithi186.stdout: create 213/34 file after the snapshot 2024-03-28T15:03:27.521 INFO:tasks.workunit.client.0.smithi186.stdout: create 213/35 file after the snapshot 2024-03-28T15:03:27.523 INFO:tasks.workunit.client.0.smithi186.stdout: create 213/36 file after the snapshot 2024-03-28T15:03:27.525 INFO:tasks.workunit.client.0.smithi186.stdout: create 213/37 file after the snapshot 2024-03-28T15:03:27.527 INFO:tasks.workunit.client.0.smithi186.stdout: create 213/38 file after the snapshot 2024-03-28T15:03:27.529 INFO:tasks.workunit.client.0.smithi186.stdout: create 213/39 file after the snapshot 2024-03-28T15:03:27.530 INFO:tasks.workunit.client.0.smithi186.stdout: create 214/.snap/snap-subdir-test 2024-03-28T15:03:27.550 INFO:tasks.workunit.client.0.smithi186.stdout: create 214/30 file after the snapshot 2024-03-28T15:03:27.551 INFO:tasks.workunit.client.0.smithi186.stdout: create 214/31 file after the snapshot 2024-03-28T15:03:27.553 INFO:tasks.workunit.client.0.smithi186.stdout: create 214/32 file after the snapshot 2024-03-28T15:03:27.555 INFO:tasks.workunit.client.0.smithi186.stdout: create 214/33 file after the snapshot 2024-03-28T15:03:27.557 INFO:tasks.workunit.client.0.smithi186.stdout: create 214/34 file after the snapshot 2024-03-28T15:03:27.559 INFO:tasks.workunit.client.0.smithi186.stdout: create 214/35 file after the snapshot 2024-03-28T15:03:27.561 INFO:tasks.workunit.client.0.smithi186.stdout: create 214/36 file after the snapshot 2024-03-28T15:03:27.566 INFO:tasks.workunit.client.0.smithi186.stdout: create 214/37 file after the snapshot 2024-03-28T15:03:27.566 INFO:tasks.workunit.client.0.smithi186.stdout: create 214/38 file after the snapshot 2024-03-28T15:03:27.566 INFO:tasks.workunit.client.0.smithi186.stdout: create 214/39 file after the snapshot 2024-03-28T15:03:27.568 INFO:tasks.workunit.client.0.smithi186.stdout: create 215/.snap/snap-subdir-test 2024-03-28T15:03:27.584 INFO:tasks.workunit.client.0.smithi186.stdout: create 215/30 file after the snapshot 2024-03-28T15:03:27.586 INFO:tasks.workunit.client.0.smithi186.stdout: create 215/31 file after the snapshot 2024-03-28T15:03:27.588 INFO:tasks.workunit.client.0.smithi186.stdout: create 215/32 file after the snapshot 2024-03-28T15:03:27.589 INFO:tasks.workunit.client.0.smithi186.stdout: create 215/33 file after the snapshot 2024-03-28T15:03:27.591 INFO:tasks.workunit.client.0.smithi186.stdout: create 215/34 file after the snapshot 2024-03-28T15:03:27.593 INFO:tasks.workunit.client.0.smithi186.stdout: create 215/35 file after the snapshot 2024-03-28T15:03:27.594 INFO:tasks.workunit.client.0.smithi186.stdout: create 215/36 file after the snapshot 2024-03-28T15:03:27.596 INFO:tasks.workunit.client.0.smithi186.stdout: create 215/37 file after the snapshot 2024-03-28T15:03:27.597 INFO:tasks.workunit.client.0.smithi186.stdout: create 215/38 file after the snapshot 2024-03-28T15:03:27.599 INFO:tasks.workunit.client.0.smithi186.stdout: create 215/39 file after the snapshot 2024-03-28T15:03:27.601 INFO:tasks.workunit.client.0.smithi186.stdout: create 216/.snap/snap-subdir-test 2024-03-28T15:03:27.616 INFO:tasks.workunit.client.0.smithi186.stdout: create 216/30 file after the snapshot 2024-03-28T15:03:27.618 INFO:tasks.workunit.client.0.smithi186.stdout: create 216/31 file after the snapshot 2024-03-28T15:03:27.620 INFO:tasks.workunit.client.0.smithi186.stdout: create 216/32 file after the snapshot 2024-03-28T15:03:27.622 INFO:tasks.workunit.client.0.smithi186.stdout: create 216/33 file after the snapshot 2024-03-28T15:03:27.624 INFO:tasks.workunit.client.0.smithi186.stdout: create 216/34 file after the snapshot 2024-03-28T15:03:27.626 INFO:tasks.workunit.client.0.smithi186.stdout: create 216/35 file after the snapshot 2024-03-28T15:03:27.628 INFO:tasks.workunit.client.0.smithi186.stdout: create 216/36 file after the snapshot 2024-03-28T15:03:27.629 INFO:tasks.workunit.client.0.smithi186.stdout: create 216/37 file after the snapshot 2024-03-28T15:03:27.631 INFO:tasks.workunit.client.0.smithi186.stdout: create 216/38 file after the snapshot 2024-03-28T15:03:27.633 INFO:tasks.workunit.client.0.smithi186.stdout: create 216/39 file after the snapshot 2024-03-28T15:03:27.635 INFO:tasks.workunit.client.0.smithi186.stdout: create 217/.snap/snap-subdir-test 2024-03-28T15:03:27.650 INFO:tasks.workunit.client.0.smithi186.stdout: create 217/30 file after the snapshot 2024-03-28T15:03:27.651 INFO:tasks.workunit.client.0.smithi186.stdout: create 217/31 file after the snapshot 2024-03-28T15:03:27.653 INFO:tasks.workunit.client.0.smithi186.stdout: create 217/32 file after the snapshot 2024-03-28T15:03:27.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.654+0000 7fd56cac9640 1 -- 172.21.15.186:0/2069722342 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd56806f870 con 0x7fd5681571a0 2024-03-28T15:03:27.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.654+0000 7fd55effd640 1 -- 172.21.15.186:0/2069722342 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6844 (secure 0 0 0) 0x7fd560060e20 con 0x7fd5681571a0 2024-03-28T15:03:27.655 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:27.655 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:03:27.655 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 13 2024-03-28T15:03:27.655 INFO:tasks.workunit.client.0.smithi186.stdout: create 217/33 file after the snapshot 2024-03-28T15:03:27.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.656+0000 7fd56cac9640 1 -- 172.21.15.186:0/2069722342 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd54806c560 msgr2=0x7fd54806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:27.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.656+0000 7fd56cac9640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd54806c560 0x7fd54806ea20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd558036000 tx=0x7fd55803b040 comp rx=0 tx=0).stop 2024-03-28T15:03:27.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.656+0000 7fd56cac9640 1 -- 172.21.15.186:0/2069722342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5681571a0 msgr2=0x7fd5681500d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:27.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.656+0000 7fd56cac9640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5681571a0 0x7fd5681500d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd56000e140 tx=0x7fd56000e600 comp rx=0 tx=0).stop 2024-03-28T15:03:27.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.657+0000 7fd56cac9640 1 -- 172.21.15.186:0/2069722342 shutdown_connections 2024-03-28T15:03:27.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.657+0000 7fd56cac9640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd54806c560 0x7fd54806ea20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.657+0000 7fd56cac9640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5681571a0 0x7fd5681500d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.657+0000 7fd56cac9640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd568072410 0x7fd56814fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.657+0000 7fd56cac9640 1 --2- 172.21.15.186:0/2069722342 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd56806f410 0x7fd56814e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.657+0000 7fd56cac9640 1 -- 172.21.15.186:0/2069722342 >> 172.21.15.186:0/2069722342 conn(0x7fd5680cd360 msgr2=0x7fd568157580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:27.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.657+0000 7fd56cac9640 1 -- 172.21.15.186:0/2069722342 shutdown_connections 2024-03-28T15:03:27.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.657+0000 7fd56cac9640 1 -- 172.21.15.186:0/2069722342 wait complete. 2024-03-28T15:03:27.658 INFO:tasks.workunit.client.0.smithi186.stdout: create 217/34 file after the snapshot 2024-03-28T15:03:27.658 INFO:tasks.workunit.client.0.smithi186.stdout: create 217/35 file after the snapshot 2024-03-28T15:03:27.660 INFO:tasks.workunit.client.0.smithi186.stdout: create 217/36 file after the snapshot 2024-03-28T15:03:27.662 INFO:tasks.workunit.client.0.smithi186.stdout: create 217/37 file after the snapshot 2024-03-28T15:03:27.664 INFO:tasks.workunit.client.0.smithi186.stdout: create 217/38 file after the snapshot 2024-03-28T15:03:27.666 INFO:tasks.workunit.client.0.smithi186.stdout: create 217/39 file after the snapshot 2024-03-28T15:03:27.668 INFO:tasks.workunit.client.0.smithi186.stdout: create 218/.snap/snap-subdir-test 2024-03-28T15:03:27.669 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2024-03-28T15:03:27.670 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T15:03:27.670 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:03:27.681 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T15:03:27.681 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:27.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.809+0000 7fb64d0a4640 1 Processor -- start 2024-03-28T15:03:27.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.809+0000 7fb64d0a4640 1 -- start start 2024-03-28T15:03:27.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb64d0a4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6480b1e30 0x7fb6480712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb64d0a4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb64806caa0 con 0x7fb648151990 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb64d0a4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb64806cd80 con 0x7fb6480c2d00 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb64d0a4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb6480c1da0 con 0x7fb6480b1e30 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb646d9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fb648151990 0x7fb6480c2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:58672/0 (socket says 172.21.15.186:58672) 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb646d9d640 1 -- 172.21.15.186:0/461850972 learned_addr learned my addr 172.21.15.186:0/461850972 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb64659c640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6480b1e30 0x7fb6480712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701545341 0 0) 0x7fb64806caa0 con 0x7fb648151990 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb630003520 con 0x7fb648151990 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 630727243 0 0) 0x7fb630003520 con 0x7fb648151990 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6480b1e30 msgr2=0x7fb6480712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb645d9b640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6480b1e30 0x7fb6480712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 >> v1:172.21.15.191:6789/0 conn(0x7fb6480c2d00 legacy=0x7fb6480717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6480c1340 con 0x7fb648151990 2024-03-28T15:03:27.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7fb6380019f0 con 0x7fb648151990 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.810+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 >> v1:172.21.15.186:6789/0 conn(0x7fb648151990 legacy=0x7fb6480c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb645d9b640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 0x7fb630003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb645d9b640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6480b1e30 0x7fb630004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb645d9b640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb630004d40 0x7fb6300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb6480c1da0 con 0x7fb6300039e0 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb63000b9c0 con 0x7fb630004d40 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb63000bbc0 con 0x7fb6480b1e30 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb646d9d640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 0x7fb630003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb64659c640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6480b1e30 0x7fb630004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb64759e640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb630004d40 0x7fb6300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb646d9d640 1 -- 172.21.15.186:0/461850972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6480b1e30 msgr2=0x7fb630004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb646d9d640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6480b1e30 0x7fb630004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb646d9d640 1 -- 172.21.15.186:0/461850972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb630004d40 msgr2=0x7fb6300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb646d9d640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb630004d40 0x7fb6300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.811+0000 7fb646d9d640 1 -- 172.21.15.186:0/461850972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb630003520 con 0x7fb6300039e0 2024-03-28T15:03:27.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.812+0000 7fb646d9d640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 0x7fb630003dc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb63800a920 tx=0x7fb630005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ced2bbc5a28712 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:27.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.812+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63800a4c0 con 0x7fb6300039e0 2024-03-28T15:03:27.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.812+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb638034650 con 0x7fb6300039e0 2024-03-28T15:03:27.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.812+0000 7fb645d9b640 1 -- 172.21.15.186:0/461850972 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63802ce60 con 0x7fb6300039e0 2024-03-28T15:03:27.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.812+0000 7fb64d0a4640 1 -- 172.21.15.186:0/461850972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 msgr2=0x7fb630003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:27.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.812+0000 7fb64d0a4640 1 --2- 172.21.15.186:0/461850972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 0x7fb630003dc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb63800a920 tx=0x7fb630005830 comp rx=0 tx=0).stop 2024-03-28T15:03:27.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.812+0000 7fb64759e640 1 -- 172.21.15.186:0/461850972 reap_dead start 2024-03-28T15:03:27.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.812+0000 7fb64d0a4640 1 -- 172.21.15.186:0/461850972 shutdown_connections 2024-03-28T15:03:27.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.812+0000 7fb64d0a4640 1 -- 172.21.15.186:0/461850972 >> 172.21.15.186:0/461850972 conn(0x7fb6480c8a20 msgr2=0x7fb6480c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:27.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.812+0000 7fb64d0a4640 1 -- 172.21.15.186:0/461850972 shutdown_connections 2024-03-28T15:03:27.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.812+0000 7fb64d0a4640 1 -- 172.21.15.186:0/461850972 wait complete. 2024-03-28T15:03:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.814+0000 7fb64d0a4640 1 Processor -- start 2024-03-28T15:03:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.814+0000 7fb64d0a4640 1 -- start start 2024-03-28T15:03:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.814+0000 7fb64d0a4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 0x7fb64815ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.814+0000 7fb64d0a4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6481e5490 0x7fb64814cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.814+0000 7fb64d0a4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6481417b0 0x7fb648141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb64d0a4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb6480c2940 con 0x7fb6300039e0 2024-03-28T15:03:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb64d0a4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb6480c1900 con 0x7fb6481417b0 2024-03-28T15:03:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb64d0a4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb6480c2ac0 con 0x7fb6481e5490 2024-03-28T15:03:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb646d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 0x7fb64815ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb646d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 0x7fb64815ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:35314/0 (socket says 172.21.15.186:35314) 2024-03-28T15:03:27.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb646d9d640 1 -- 172.21.15.186:0/2036020760 learned_addr learned my addr 172.21.15.186:0/2036020760 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb64659c640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6481e5490 0x7fb64814cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb64759e640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6481417b0 0x7fb648141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb646d9d640 1 -- 172.21.15.186:0/2036020760 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6481e5490 msgr2=0x7fb64814cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb646d9d640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6481e5490 0x7fb64814cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb646d9d640 1 -- 172.21.15.186:0/2036020760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6481417b0 msgr2=0x7fb648141c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb646d9d640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6481417b0 0x7fb648141c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb646d9d640 1 -- 172.21.15.186:0/2036020760 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6481f2500 con 0x7fb6300039e0 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb64659c640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6481e5490 0x7fb64814cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb64759e640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6481417b0 0x7fb648141c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb646d9d640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 0x7fb64815ad80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb638001cf0 tx=0x7fb638001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb62f7fe640 1 -- 172.21.15.186:0/2036020760 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb638034a00 con 0x7fb6300039e0 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb62f7fe640 1 -- 172.21.15.186:0/2036020760 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb63803a040 con 0x7fb6300039e0 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb64d0a4640 1 -- 172.21.15.186:0/2036020760 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb648158850 con 0x7fb6300039e0 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb62f7fe640 1 -- 172.21.15.186:0/2036020760 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb638034de0 con 0x7fb6300039e0 2024-03-28T15:03:27.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.815+0000 7fb64d0a4640 1 -- 172.21.15.186:0/2036020760 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb648158e10 con 0x7fb6300039e0 2024-03-28T15:03:27.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.816+0000 7fb64d0a4640 1 -- 172.21.15.186:0/2036020760 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb648069500 con 0x7fb6300039e0 2024-03-28T15:03:27.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.817+0000 7fb62f7fe640 1 -- 172.21.15.186:0/2036020760 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb63803f420 con 0x7fb6300039e0 2024-03-28T15:03:27.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.817+0000 7fb62f7fe640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb63406c530 0x7fb63406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:27.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.817+0000 7fb64659c640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb63406c530 0x7fb63406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:27.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.817+0000 7fb62f7fe640 1 -- 172.21.15.186:0/2036020760 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5939+0+0 (secure 0 0 0) 0x7fb6380ad9a0 con 0x7fb6300039e0 2024-03-28T15:03:27.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.817+0000 7fb64659c640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb63406c530 0x7fb63406e9f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb64814e7f0 tx=0x7fb63c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:27.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:27.821+0000 7fb62f7fe640 1 -- 172.21.15.186:0/2036020760 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb63807c040 con 0x7fb6300039e0 2024-03-28T15:03:28.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.033+0000 7fb64d0a4640 1 -- 172.21.15.186:0/2036020760 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6480b1a00 con 0x7fb6300039e0 2024-03-28T15:03:28.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.034+0000 7fb62f7fe640 1 -- 172.21.15.186:0/2036020760 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6844 (secure 0 0 0) 0x7fb63807b940 con 0x7fb6300039e0 2024-03-28T15:03:28.034 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:28.034 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:03:28.035 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 13 2024-03-28T15:03:28.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 -- 172.21.15.186:0/2036020760 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb63406c530 msgr2=0x7fb63406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:28.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb63406c530 0x7fb63406e9f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb64814e7f0 tx=0x7fb63c008040 comp rx=0 tx=0).stop 2024-03-28T15:03:28.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 -- 172.21.15.186:0/2036020760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 msgr2=0x7fb64815ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:28.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 0x7fb64815ad80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb638001cf0 tx=0x7fb638001dc0 comp rx=0 tx=0).stop 2024-03-28T15:03:28.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 -- 172.21.15.186:0/2036020760 shutdown_connections 2024-03-28T15:03:28.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb63406c530 0x7fb63406e9f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:28.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6481417b0 0x7fb648141c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:28.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6481e5490 0x7fb64814cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:28.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 --2- 172.21.15.186:0/2036020760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6300039e0 0x7fb64815ad80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:28.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 -- 172.21.15.186:0/2036020760 >> 172.21.15.186:0/2036020760 conn(0x7fb6480c8a20 msgr2=0x7fb6480736a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:28.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 -- 172.21.15.186:0/2036020760 shutdown_connections 2024-03-28T15:03:28.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:28.036+0000 7fb64d0a4640 1 -- 172.21.15.186:0/2036020760 wait complete. 2024-03-28T15:03:28.047 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:03:30.049 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:30.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.161+0000 7fd6f56fb640 1 Processor -- start 2024-03-28T15:03:30.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.161+0000 7fd6f56fb640 1 -- start start 2024-03-28T15:03:30.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.161+0000 7fd6f56fb640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd6f00cb180 0x7fd6f0166220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:30.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.161+0000 7fd6f56fb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f0072b30 con 0x7fd6f0154dc0 2024-03-28T15:03:30.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.161+0000 7fd6f56fb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f00cb7a0 con 0x7fd6f00c52f0 2024-03-28T15:03:30.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.161+0000 7fd6f56fb640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f015eb70 con 0x7fd6f00cb180 2024-03-28T15:03:30.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.162+0000 7fd6eeffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd6f0154dc0 0x7fd6f0072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:56256/0 (socket says 172.21.15.186:56256) 2024-03-28T15:03:30.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.162+0000 7fd6eeffd640 1 -- 172.21.15.186:0/2721392704 learned_addr learned my addr 172.21.15.186:0/2721392704 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:03:30.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.162+0000 7fd6ef7fe640 1 --2- 172.21.15.186:0/2721392704 >> v2:172.21.15.191:3300/0 conn(0x7fd6f00cb180 0x7fd6f0166220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:30.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.162+0000 7fd6edffb640 1 -- 172.21.15.186:0/2721392704 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422922687 0 0) 0x7fd6f0072b30 con 0x7fd6f0154dc0 2024-03-28T15:03:30.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.162+0000 7fd6edffb640 1 -- 172.21.15.186:0/2721392704 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6d8003520 con 0x7fd6f0154dc0 2024-03-28T15:03:30.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.162+0000 7fd6edffb640 1 -- 172.21.15.186:0/2721392704 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1225033986 0 0) 0x7fd6f00cb7a0 con 0x7fd6f00c52f0 2024-03-28T15:03:30.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.162+0000 7fd6edffb640 1 -- 172.21.15.186:0/2721392704 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6f0072b30 con 0x7fd6f00c52f0 2024-03-28T15:03:30.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.162+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/2721392704 >> v1:172.21.15.191:6789/0 conn(0x7fd6f00c52f0 legacy=0x7fd6f00caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:30.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.162+0000 7fd6edffb640 1 -- 172.21.15.186:0/2721392704 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3535334357 0 0) 0x7fd6d8003520 con 0x7fd6f0154dc0 2024-03-28T15:03:30.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.162+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/2721392704 >> v1:172.21.15.186:6789/0 conn(0x7fd6f0154dc0 legacy=0x7fd6f0072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:30.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.162+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/2721392704 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f0071c20 con 0x7fd6f00cb180 2024-03-28T15:03:30.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.163+0000 7fd6ef7fe640 1 --2- 172.21.15.186:0/2721392704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00cb180 0x7fd6f0166220 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009d90 tx=0x7fd6e802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1033b064a234bcd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:30.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.163+0000 7fd6edffb640 1 -- 172.21.15.186:0/2721392704 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8039070 con 0x7fd6f00cb180 2024-03-28T15:03:30.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.163+0000 7fd6edffb640 1 -- 172.21.15.186:0/2721392704 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e802c940 con 0x7fd6f00cb180 2024-03-28T15:03:30.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.163+0000 7fd6edffb640 1 -- 172.21.15.186:0/2721392704 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0001670 con 0x7fd6f00cb180 2024-03-28T15:03:30.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.164+0000 7fd6f56fb640 1 -- 172.21.15.186:0/2721392704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00cb180 msgr2=0x7fd6f0166220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:30.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.164+0000 7fd6f56fb640 1 --2- 172.21.15.186:0/2721392704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00cb180 0x7fd6f0166220 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009d90 tx=0x7fd6e802be10 comp rx=0 tx=0).stop 2024-03-28T15:03:30.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.164+0000 7fd6f56fb640 1 -- 172.21.15.186:0/2721392704 shutdown_connections 2024-03-28T15:03:30.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.164+0000 7fd6f56fb640 1 --2- 172.21.15.186:0/2721392704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00cb180 0x7fd6f0166220 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:30.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.164+0000 7fd6f56fb640 1 -- 172.21.15.186:0/2721392704 >> 172.21.15.186:0/2721392704 conn(0x7fd6f00d00f0 msgr2=0x7fd6f00d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:30.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.164+0000 7fd6f56fb640 1 -- 172.21.15.186:0/2721392704 shutdown_connections 2024-03-28T15:03:30.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.164+0000 7fd6f56fb640 1 -- 172.21.15.186:0/2721392704 wait complete. 2024-03-28T15:03:30.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.165+0000 7fd6f56fb640 1 Processor -- start 2024-03-28T15:03:30.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.165+0000 7fd6f56fb640 1 -- start start 2024-03-28T15:03:30.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6f56fb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f00c52f0 0x7fd6f01f1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:30.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6f56fb640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00cb180 0x7fd6f00cf270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:30.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6f56fb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154dc0 0x7fd6f00c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:30.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6f56fb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f015e840 con 0x7fd6f0154dc0 2024-03-28T15:03:30.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6f56fb640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f00cc7e0 con 0x7fd6f00cb180 2024-03-28T15:03:30.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6f56fb640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f00c5840 con 0x7fd6f00c52f0 2024-03-28T15:03:30.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6ef7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154dc0 0x7fd6f00c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:30.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6ef7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154dc0 0x7fd6f00c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41550/0 (socket says 172.21.15.186:41550) 2024-03-28T15:03:30.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/1346707906 learned_addr learned my addr 172.21.15.186:0/1346707906 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:30.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6ee7fc640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00cb180 0x7fd6f00cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:30.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6eeffd640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f00c52f0 0x7fd6f01f1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:30.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/1346707906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f00c52f0 msgr2=0x7fd6f01f1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6ef7fe640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f00c52f0 0x7fd6f01f1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/1346707906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00cb180 msgr2=0x7fd6f00cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6ef7fe640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00cb180 0x7fd6f00cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/1346707906 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f01e4e30 con 0x7fd6f0154dc0 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6ee7fc640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00cb180 0x7fd6f00cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6ef7fe640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154dc0 0x7fd6f00c85a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009eb0 tx=0x7fd6e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.166+0000 7fd6d77fe640 1 -- 172.21.15.186:0/1346707906 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8039050 con 0x7fd6f0154dc0 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.167+0000 7fd6d77fe640 1 -- 172.21.15.186:0/1346707906 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e8045070 con 0x7fd6f0154dc0 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.167+0000 7fd6f56fb640 1 -- 172.21.15.186:0/1346707906 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f015dec0 con 0x7fd6f0154dc0 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.167+0000 7fd6d77fe640 1 -- 172.21.15.186:0/1346707906 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8009450 con 0x7fd6f0154dc0 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.167+0000 7fd6f56fb640 1 -- 172.21.15.186:0/1346707906 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f015bdf0 con 0x7fd6f0154dc0 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.168+0000 7fd6f56fb640 1 -- 172.21.15.186:0/1346707906 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f015cb90 con 0x7fd6f0154dc0 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.168+0000 7fd6d77fe640 1 -- 172.21.15.186:0/1346707906 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd6e8040690 con 0x7fd6f0154dc0 2024-03-28T15:03:30.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.168+0000 7fd6d77fe640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 0x7fd6dc06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:30.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.169+0000 7fd6d77fe640 1 -- 172.21.15.186:0/1346707906 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5939+0+0 (secure 0 0 0) 0x7fd6e80ada50 con 0x7fd6f0154dc0 2024-03-28T15:03:30.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.169+0000 7fd6eeffd640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 0x7fd6dc06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:30.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.169+0000 7fd6eeffd640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 0x7fd6dc06ea20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0000fd0 tx=0x7fd6e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:30.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.173+0000 7fd6d77fe640 1 -- 172.21.15.186:0/1346707906 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd6f015cb90 con 0x7fd6f0154dc0 2024-03-28T15:03:30.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.386+0000 7fd6f56fb640 1 -- 172.21.15.186:0/1346707906 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6f0072d00 con 0x7fd6f0154dc0 2024-03-28T15:03:30.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.386+0000 7fd6d77fe640 1 -- 172.21.15.186:0/1346707906 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6844 (secure 0 0 0) 0x7fd6e804e020 con 0x7fd6f0154dc0 2024-03-28T15:03:30.387 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:30.387 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:03:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 13 2024-03-28T15:03:30.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.388+0000 7fd6f56fb640 1 -- 172.21.15.186:0/1346707906 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 msgr2=0x7fd6dc06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:30.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.388+0000 7fd6f56fb640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 0x7fd6dc06ea20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0000fd0 tx=0x7fd6e0009040 comp rx=0 tx=0).stop 2024-03-28T15:03:30.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.388+0000 7fd6f56fb640 1 -- 172.21.15.186:0/1346707906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154dc0 msgr2=0x7fd6f00c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:30.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.388+0000 7fd6f56fb640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154dc0 0x7fd6f00c85a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009eb0 tx=0x7fd6e8004a40 comp rx=0 tx=0).stop 2024-03-28T15:03:30.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.388+0000 7fd6f56fb640 1 -- 172.21.15.186:0/1346707906 shutdown_connections 2024-03-28T15:03:30.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.388+0000 7fd6f56fb640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 0x7fd6dc06ea20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.388+0000 7fd6f56fb640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154dc0 0x7fd6f00c85a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.388+0000 7fd6f56fb640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00cb180 0x7fd6f00cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.388+0000 7fd6f56fb640 1 --2- 172.21.15.186:0/1346707906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f00c52f0 0x7fd6f01f1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.388+0000 7fd6f56fb640 1 -- 172.21.15.186:0/1346707906 >> 172.21.15.186:0/1346707906 conn(0x7fd6f00d00f0 msgr2=0x7fd6f014fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.389+0000 7fd6f56fb640 1 -- 172.21.15.186:0/1346707906 shutdown_connections 2024-03-28T15:03:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:30.389+0000 7fd6f56fb640 1 -- 172.21.15.186:0/1346707906 wait complete. 2024-03-28T15:03:30.400 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:03:32.400 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:32.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.513+0000 7f3242ae1640 1 Processor -- start 2024-03-28T15:03:32.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.513+0000 7f3242ae1640 1 -- start start 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f3242ae1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f323c152810 0x7f323c158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f3242ae1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f323c0c8090 0x7f323c0c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f3242ae1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f323c072df0 con 0x7f323c071340 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f3242ae1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f323c159500 con 0x7f323c152810 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f3242ae1640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f323c0c8660 con 0x7f323c0c8090 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f324087d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f323c152810 0x7f323c158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f324087d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f323c152810 0x7f323c158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41566/0 (socket says 172.21.15.186:41566) 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f324087d640 1 -- 172.21.15.186:0/1888153 learned_addr learned my addr 172.21.15.186:0/1888153 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f324107e640 1 --2- 172.21.15.186:0/1888153 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f323c0c8090 0x7f323c0c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f324087d640 1 -- 172.21.15.186:0/1888153 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f323c0c8090 msgr2=0x7f323c0c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f324087d640 1 --2- 172.21.15.186:0/1888153 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f323c0c8090 0x7f323c0c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f324087d640 1 -- 172.21.15.186:0/1888153 >> v1:172.21.15.191:6789/0 conn(0x7f323c071340 legacy=0x7f323c06c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:32.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.514+0000 7f324087d640 1 -- 172.21.15.186:0/1888153 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f323c071d50 con 0x7f323c152810 2024-03-28T15:03:32.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.515+0000 7f324087d640 1 --2- 172.21.15.186:0/1888153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f323c152810 0x7f323c158fc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3230009870 tx=0x7f323002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad5c04b2333a949 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:32.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.515+0000 7f323b7fe640 1 -- 172.21.15.186:0/1888153 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3230034070 con 0x7f323c152810 2024-03-28T15:03:32.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.515+0000 7f323b7fe640 1 -- 172.21.15.186:0/1888153 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3230030840 con 0x7f323c152810 2024-03-28T15:03:32.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.515+0000 7f323b7fe640 1 -- 172.21.15.186:0/1888153 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3230030c10 con 0x7f323c152810 2024-03-28T15:03:32.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.516+0000 7f3242ae1640 1 -- 172.21.15.186:0/1888153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f323c152810 msgr2=0x7f323c158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:32.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.516+0000 7f3242ae1640 1 --2- 172.21.15.186:0/1888153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f323c152810 0x7f323c158fc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3230009870 tx=0x7f323002b980 comp rx=0 tx=0).stop 2024-03-28T15:03:32.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.516+0000 7f3242ae1640 1 -- 172.21.15.186:0/1888153 shutdown_connections 2024-03-28T15:03:32.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.516+0000 7f3242ae1640 1 --2- 172.21.15.186:0/1888153 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f323c0c8090 0x7f323c0c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:32.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.516+0000 7f3242ae1640 1 --2- 172.21.15.186:0/1888153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f323c152810 0x7f323c158fc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:32.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.516+0000 7f3242ae1640 1 -- 172.21.15.186:0/1888153 >> 172.21.15.186:0/1888153 conn(0x7f323c0c8a20 msgr2=0x7f323c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:32.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.516+0000 7f3242ae1640 1 -- 172.21.15.186:0/1888153 shutdown_connections 2024-03-28T15:03:32.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.516+0000 7f3242ae1640 1 -- 172.21.15.186:0/1888153 wait complete. 2024-03-28T15:03:32.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.517+0000 7f3242ae1640 1 Processor -- start 2024-03-28T15:03:32.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.517+0000 7f3242ae1640 1 -- start start 2024-03-28T15:03:32.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f3242ae1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f323c071340 0x7f323c1454f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:32.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f3242ae1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f323c0c8090 0x7f323c14ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:32.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f3242ae1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f323c152810 0x7f323c14bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:32.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f3242ae1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f323c0725f0 con 0x7f323c152810 2024-03-28T15:03:32.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f3242ae1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f323c072310 con 0x7f323c0c8090 2024-03-28T15:03:32.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f3242ae1640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f323c0728d0 con 0x7f323c071340 2024-03-28T15:03:32.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f324107e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f323c152810 0x7f323c14bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f324107e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f323c152810 0x7f323c14bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41570/0 (socket says 172.21.15.186:41570) 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f324107e640 1 -- 172.21.15.186:0/845647526 learned_addr learned my addr 172.21.15.186:0/845647526 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f324087d640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f323c071340 0x7f323c1454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f323bfff640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f323c0c8090 0x7f323c14ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f324107e640 1 -- 172.21.15.186:0/845647526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f323c071340 msgr2=0x7f323c1454f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f324107e640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f323c071340 0x7f323c1454f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f324107e640 1 -- 172.21.15.186:0/845647526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f323c0c8090 msgr2=0x7f323c14ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f324107e640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f323c0c8090 0x7f323c14ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f324107e640 1 -- 172.21.15.186:0/845647526 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f323c1e44f0 con 0x7f323c152810 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f324087d640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f323c071340 0x7f323c1454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f324107e640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f323c152810 0x7f323c14bfc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f323400efc0 tx=0x7f323400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.518+0000 7f32397fa640 1 -- 172.21.15.186:0/845647526 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3234018020 con 0x7f323c152810 2024-03-28T15:03:32.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.519+0000 7f32397fa640 1 -- 172.21.15.186:0/845647526 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3234014480 con 0x7f323c152810 2024-03-28T15:03:32.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.519+0000 7f3242ae1640 1 -- 172.21.15.186:0/845647526 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f323c14e920 con 0x7f323c152810 2024-03-28T15:03:32.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.519+0000 7f32397fa640 1 -- 172.21.15.186:0/845647526 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3234014870 con 0x7f323c152810 2024-03-28T15:03:32.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.519+0000 7f3242ae1640 1 -- 172.21.15.186:0/845647526 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f323c14eee0 con 0x7f323c152810 2024-03-28T15:03:32.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.520+0000 7f3242ae1640 1 -- 172.21.15.186:0/845647526 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f323c069500 con 0x7f323c152810 2024-03-28T15:03:32.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.520+0000 7f32397fa640 1 -- 172.21.15.186:0/845647526 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3234005d10 con 0x7f323c152810 2024-03-28T15:03:32.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.520+0000 7f32397fa640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322006c560 0x7f322006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:32.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.521+0000 7f32397fa640 1 -- 172.21.15.186:0/845647526 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5939+0+0 (secure 0 0 0) 0x7f323408c600 con 0x7f323c152810 2024-03-28T15:03:32.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.521+0000 7f324087d640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322006c560 0x7f322006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:32.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.521+0000 7f324087d640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322006c560 0x7f322006ea20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3230036000 tx=0x7f323003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:32.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.525+0000 7f32397fa640 1 -- 172.21.15.186:0/845647526 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3234022030 con 0x7f323c152810 2024-03-28T15:03:32.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.737+0000 7f3242ae1640 1 -- 172.21.15.186:0/845647526 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f323c148e70 con 0x7f323c152810 2024-03-28T15:03:32.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.737+0000 7f32397fa640 1 -- 172.21.15.186:0/845647526 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6844 (secure 0 0 0) 0x7f3234010040 con 0x7f323c152810 2024-03-28T15:03:32.738 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:32.738 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:03:32.738 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 13 2024-03-28T15:03:32.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.739+0000 7f3242ae1640 1 -- 172.21.15.186:0/845647526 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322006c560 msgr2=0x7f322006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:32.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.739+0000 7f3242ae1640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322006c560 0x7f322006ea20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3230036000 tx=0x7f323003b040 comp rx=0 tx=0).stop 2024-03-28T15:03:32.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.739+0000 7f3242ae1640 1 -- 172.21.15.186:0/845647526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f323c152810 msgr2=0x7f323c14bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:32.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.739+0000 7f3242ae1640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f323c152810 0x7f323c14bfc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f323400efc0 tx=0x7f323400c490 comp rx=0 tx=0).stop 2024-03-28T15:03:32.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.739+0000 7f3242ae1640 1 -- 172.21.15.186:0/845647526 shutdown_connections 2024-03-28T15:03:32.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.739+0000 7f3242ae1640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f322006c560 0x7f322006ea20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:32.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.739+0000 7f3242ae1640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f323c152810 0x7f323c14bfc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:32.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.739+0000 7f3242ae1640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f323c0c8090 0x7f323c14ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:32.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.739+0000 7f3242ae1640 1 --2- 172.21.15.186:0/845647526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f323c071340 0x7f323c1454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:32.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.739+0000 7f3242ae1640 1 -- 172.21.15.186:0/845647526 >> 172.21.15.186:0/845647526 conn(0x7f323c0c8a20 msgr2=0x7f323c14dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:32.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.740+0000 7f3242ae1640 1 -- 172.21.15.186:0/845647526 shutdown_connections 2024-03-28T15:03:32.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:32.740+0000 7f3242ae1640 1 -- 172.21.15.186:0/845647526 wait complete. 2024-03-28T15:03:32.752 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:03:34.753 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:34.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.863+0000 7f7b26488640 1 Processor -- start 2024-03-28T15:03:34.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.863+0000 7f7b26488640 1 -- start start 2024-03-28T15:03:34.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.863+0000 7f7b26488640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7b20154bd0 0x7f7b2015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:34.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.863+0000 7f7b26488640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b200c53c0 0x7f7b200c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:34.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.863+0000 7f7b26488640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7b200c57e0 0x7f7b200cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:34.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.863+0000 7f7b26488640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f7b200c5ca0 con 0x7f7b20154bd0 2024-03-28T15:03:34.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.863+0000 7f7b26488640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f7b2015b080 con 0x7f7b200c57e0 2024-03-28T15:03:34.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.863+0000 7f7b26488640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7b2015b360 con 0x7f7b200c53c0 2024-03-28T15:03:34.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b1ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7b20154bd0 0x7f7b2015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b1ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7b20154bd0 0x7f7b2015a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41574/0 (socket says 172.21.15.186:41574) 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b1ffff640 1 -- 172.21.15.186:0/1657184671 learned_addr learned my addr 172.21.15.186:0/1657184671 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b1f7fe640 1 --2- 172.21.15.186:0/1657184671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b200c53c0 0x7f7b200c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b24a25640 1 --2- 172.21.15.186:0/1657184671 >> v2:172.21.15.191:3300/0 conn(0x7f7b200c57e0 0x7f7b200cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b1ffff640 1 -- 172.21.15.186:0/1657184671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b200c53c0 msgr2=0x7f7b200c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b1ffff640 1 --2- 172.21.15.186:0/1657184671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b200c53c0 0x7f7b200c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b1ffff640 1 -- 172.21.15.186:0/1657184671 >> v2:172.21.15.191:3300/0 conn(0x7f7b200c57e0 msgr2=0x7f7b200cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b1ffff640 1 --2- 172.21.15.186:0/1657184671 >> v2:172.21.15.191:3300/0 conn(0x7f7b200c57e0 0x7f7b200cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b1ffff640 1 -- 172.21.15.186:0/1657184671 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b2015b640 con 0x7f7b20154bd0 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b1f7fe640 1 --2- 172.21.15.186:0/1657184671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b200c53c0 0x7f7b200c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.864+0000 7f7b24a25640 1 --2- 172.21.15.186:0/1657184671 >> v2:172.21.15.191:3300/0 conn(0x7f7b200c57e0 0x7f7b200cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.865+0000 7f7b1ffff640 1 --2- 172.21.15.186:0/1657184671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b20154bd0 0x7f7b2015a7d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7b2015ee60 tx=0x7f7b1002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60ffd8a6f608530 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.865+0000 7f7b1effd640 1 -- 172.21.15.186:0/1657184671 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b10034070 con 0x7f7b20154bd0 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.865+0000 7f7b1effd640 1 -- 172.21.15.186:0/1657184671 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b10030860 con 0x7f7b20154bd0 2024-03-28T15:03:34.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.865+0000 7f7b1effd640 1 -- 172.21.15.186:0/1657184671 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b10030c30 con 0x7f7b20154bd0 2024-03-28T15:03:34.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.865+0000 7f7b26488640 1 -- 172.21.15.186:0/1657184671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b20154bd0 msgr2=0x7f7b2015a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:34.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.865+0000 7f7b26488640 1 --2- 172.21.15.186:0/1657184671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b20154bd0 0x7f7b2015a7d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7b2015ee60 tx=0x7f7b1002b9a0 comp rx=0 tx=0).stop 2024-03-28T15:03:34.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.866+0000 7f7b26488640 1 -- 172.21.15.186:0/1657184671 shutdown_connections 2024-03-28T15:03:34.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.866+0000 7f7b26488640 1 --2- 172.21.15.186:0/1657184671 >> v2:172.21.15.191:3300/0 conn(0x7f7b200c57e0 0x7f7b200cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.866+0000 7f7b26488640 1 --2- 172.21.15.186:0/1657184671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b200c53c0 0x7f7b200c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.866+0000 7f7b26488640 1 --2- 172.21.15.186:0/1657184671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b20154bd0 0x7f7b2015a7d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.866+0000 7f7b26488640 1 -- 172.21.15.186:0/1657184671 >> 172.21.15.186:0/1657184671 conn(0x7f7b200731f0 msgr2=0x7f7b20073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.866+0000 7f7b26488640 1 -- 172.21.15.186:0/1657184671 shutdown_connections 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.866+0000 7f7b26488640 1 -- 172.21.15.186:0/1657184671 wait complete. 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.867+0000 7f7b26488640 1 Processor -- start 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.867+0000 7f7b26488640 1 -- start start 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b26488640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b200c53c0 0x7f7b20157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b26488640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b200c57e0 0x7f7b20157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b26488640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b20154bd0 0x7f7b2014a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b26488640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7b200cc390 con 0x7f7b200c53c0 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b26488640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7b200c61d0 con 0x7f7b200c57e0 2024-03-28T15:03:34.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b26488640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7b2015ad10 con 0x7f7b20154bd0 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b200c53c0 0x7f7b20157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b200c53c0 0x7f7b20157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41584/0 (socket says 172.21.15.186:41584) 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1ffff640 1 -- 172.21.15.186:0/3675572699 learned_addr learned my addr 172.21.15.186:0/3675572699 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1ffff640 1 -- 172.21.15.186:0/3675572699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b20154bd0 msgr2=0x7f7b2014a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b24a25640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b20154bd0 0x7f7b2014a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1f7fe640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b200c57e0 0x7f7b20157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1ffff640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b20154bd0 0x7f7b2014a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1ffff640 1 -- 172.21.15.186:0/3675572699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b200c57e0 msgr2=0x7f7b20157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1ffff640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b200c57e0 0x7f7b20157860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1ffff640 1 -- 172.21.15.186:0/3675572699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b201f0f10 con 0x7f7b200c53c0 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b24a25640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b20154bd0 0x7f7b2014a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1ffff640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b200c53c0 0x7f7b20157320 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7b1002b9e0 tx=0x7f7b100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1cff9640 1 -- 172.21.15.186:0/3675572699 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b10034040 con 0x7f7b200c53c0 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1cff9640 1 -- 172.21.15.186:0/3675572699 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b10039d90 con 0x7f7b200c53c0 2024-03-28T15:03:34.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b26488640 1 -- 172.21.15.186:0/3675572699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b201f11f0 con 0x7f7b200c53c0 2024-03-28T15:03:34.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b1cff9640 1 -- 172.21.15.186:0/3675572699 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b10043610 con 0x7f7b200c53c0 2024-03-28T15:03:34.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.868+0000 7f7b26488640 1 -- 172.21.15.186:0/3675572699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b201f17b0 con 0x7f7b200c53c0 2024-03-28T15:03:34.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.869+0000 7f7b26488640 1 -- 172.21.15.186:0/3675572699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b200b3d60 con 0x7f7b200c53c0 2024-03-28T15:03:34.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.870+0000 7f7b1cff9640 1 -- 172.21.15.186:0/3675572699 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7b1004a060 con 0x7f7b200c53c0 2024-03-28T15:03:34.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.870+0000 7f7b1cff9640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7b0006c530 0x7f7b0006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:34.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.870+0000 7f7b1cff9640 1 -- 172.21.15.186:0/3675572699 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5939+0+0 (secure 0 0 0) 0x7f7b100ad890 con 0x7f7b200c53c0 2024-03-28T15:03:34.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.870+0000 7f7b1f7fe640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7b0006c530 0x7f7b0006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:34.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.871+0000 7f7b1f7fe640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7b0006c530 0x7f7b0006e9f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7b08006fd0 tx=0x7f7b08008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:34.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:34.875+0000 7f7b1cff9640 1 -- 172.21.15.186:0/3675572699 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7b200b3d60 con 0x7f7b200c53c0 2024-03-28T15:03:35.091 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.090+0000 7f7b26488640 1 -- 172.21.15.186:0/3675572699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b20069500 con 0x7f7b200c53c0 2024-03-28T15:03:35.091 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.090+0000 7f7b1cff9640 1 -- 172.21.15.186:0/3675572699 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6844 (secure 0 0 0) 0x7f7b1003e070 con 0x7f7b200c53c0 2024-03-28T15:03:35.092 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:35.092 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:03:35.092 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 13 2024-03-28T15:03:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.092+0000 7f7b26488640 1 -- 172.21.15.186:0/3675572699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7b0006c530 msgr2=0x7f7b0006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.092+0000 7f7b26488640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7b0006c530 0x7f7b0006e9f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7b08006fd0 tx=0x7f7b08008040 comp rx=0 tx=0).stop 2024-03-28T15:03:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.092+0000 7f7b26488640 1 -- 172.21.15.186:0/3675572699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b200c53c0 msgr2=0x7f7b20157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.092+0000 7f7b26488640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b200c53c0 0x7f7b20157320 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7b1002b9e0 tx=0x7f7b100046c0 comp rx=0 tx=0).stop 2024-03-28T15:03:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.092+0000 7f7b26488640 1 -- 172.21.15.186:0/3675572699 shutdown_connections 2024-03-28T15:03:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.092+0000 7f7b26488640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7b0006c530 0x7f7b0006e9f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.092+0000 7f7b26488640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b20154bd0 0x7f7b2014a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.092+0000 7f7b26488640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b200c57e0 0x7f7b20157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.092+0000 7f7b26488640 1 --2- 172.21.15.186:0/3675572699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b200c53c0 0x7f7b20157320 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.092+0000 7f7b26488640 1 -- 172.21.15.186:0/3675572699 >> 172.21.15.186:0/3675572699 conn(0x7f7b200731f0 msgr2=0x7f7b200d00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.093+0000 7f7b26488640 1 -- 172.21.15.186:0/3675572699 shutdown_connections 2024-03-28T15:03:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:35.093+0000 7f7b26488640 1 -- 172.21.15.186:0/3675572699 wait complete. 2024-03-28T15:03:35.105 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:03:37.106 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:37.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.217+0000 7f75d93b5640 1 Processor -- start 2024-03-28T15:03:37.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.217+0000 7f75d93b5640 1 -- start start 2024-03-28T15:03:37.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.218+0000 7f75d93b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75d41571a0 0x7f75d40c3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.218+0000 7f75d93b5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f75d4072410 0x7f75d40c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.218+0000 7f75d93b5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75d406eaf0 con 0x7f75d406f410 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.218+0000 7f75d93b5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f75d40c9070 con 0x7f75d4072410 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.218+0000 7f75d93b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f75d40c84f0 con 0x7f75d41571a0 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.218+0000 7f75d37fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f75d4072410 0x7f75d40c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.218+0000 7f75d27fc640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f75d406f410 0x7f75d415b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:56270/0 (socket says 172.21.15.186:56270) 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.218+0000 7f75d27fc640 1 -- 172.21.15.186:0/20367585 learned_addr learned my addr 172.21.15.186:0/20367585 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.219+0000 7f75d2ffd640 1 --2- 172.21.15.186:0/20367585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75d41571a0 0x7f75d40c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.219+0000 7f75d37fe640 1 -- 172.21.15.186:0/20367585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75d41571a0 msgr2=0x7f75d40c3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.219+0000 7f75d37fe640 1 --2- 172.21.15.186:0/20367585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75d41571a0 0x7f75d40c3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.219+0000 7f75d37fe640 1 -- 172.21.15.186:0/20367585 >> v1:172.21.15.186:6789/0 conn(0x7f75d406f410 legacy=0x7f75d415b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:37.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.219+0000 7f75d37fe640 1 -- 172.21.15.186:0/20367585 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d40c87d0 con 0x7f75d4072410 2024-03-28T15:03:37.222 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.219+0000 7f75d1ffb640 1 -- 172.21.15.186:0/20367585 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3754700238 0 0) 0x7f75c8001110 con 0x7f75d406f410 2024-03-28T15:03:37.222 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.219+0000 7f75d37fe640 1 --2- 172.21.15.186:0/20367585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d4072410 0x7f75d40c2b40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0097a0 tx=0x7f75cc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c3cda8a6da0894 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:37.222 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.220+0000 7f75d1ffb640 1 -- 172.21.15.186:0/20367585 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc034070 con 0x7f75d4072410 2024-03-28T15:03:37.222 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.220+0000 7f75d1ffb640 1 -- 172.21.15.186:0/20367585 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75cc030610 con 0x7f75d4072410 2024-03-28T15:03:37.222 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.220+0000 7f75d1ffb640 1 -- 172.21.15.186:0/20367585 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc0309e0 con 0x7f75d4072410 2024-03-28T15:03:37.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.220+0000 7f75d93b5640 1 -- 172.21.15.186:0/20367585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d4072410 msgr2=0x7f75d40c2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:37.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.220+0000 7f75d93b5640 1 --2- 172.21.15.186:0/20367585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d4072410 0x7f75d40c2b40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0097a0 tx=0x7f75cc02b960 comp rx=0 tx=0).stop 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.220+0000 7f75d93b5640 1 -- 172.21.15.186:0/20367585 shutdown_connections 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.220+0000 7f75d93b5640 1 --2- 172.21.15.186:0/20367585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d4072410 0x7f75d40c2b40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.220+0000 7f75d93b5640 1 --2- 172.21.15.186:0/20367585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75d41571a0 0x7f75d40c3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.220+0000 7f75d93b5640 1 -- 172.21.15.186:0/20367585 >> 172.21.15.186:0/20367585 conn(0x7f75d40cd360 msgr2=0x7f75d40cd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.220+0000 7f75d93b5640 1 -- 172.21.15.186:0/20367585 shutdown_connections 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.220+0000 7f75d93b5640 1 -- 172.21.15.186:0/20367585 wait complete. 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.222+0000 7f75d93b5640 1 Processor -- start 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.222+0000 7f75d93b5640 1 -- start start 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.222+0000 7f75d93b5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75d406f410 0x7f75d40cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.222+0000 7f75d93b5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d4072410 0x7f75d4158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.222+0000 7f75d93b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75d41571a0 0x7f75d4158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.222+0000 7f75d93b5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75d415ad90 con 0x7f75d4072410 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.222+0000 7f75d93b5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f75d40c8d90 con 0x7f75d406f410 2024-03-28T15:03:37.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.222+0000 7f75d93b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f75d415b9c0 con 0x7f75d41571a0 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.222+0000 7f75d27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d4072410 0x7f75d4158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.222+0000 7f75d27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d4072410 0x7f75d4158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41588/0 (socket says 172.21.15.186:41588) 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75d27fc640 1 -- 172.21.15.186:0/707319594 learned_addr learned my addr 172.21.15.186:0/707319594 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75d2ffd640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75d406f410 0x7f75d40cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75d37fe640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75d41571a0 0x7f75d4158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75d27fc640 1 -- 172.21.15.186:0/707319594 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75d41571a0 msgr2=0x7f75d4158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75d27fc640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75d41571a0 0x7f75d4158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75d27fc640 1 -- 172.21.15.186:0/707319594 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75d406f410 msgr2=0x7f75d40cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75d27fc640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75d406f410 0x7f75d40cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75d27fc640 1 -- 172.21.15.186:0/707319594 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d40c7820 con 0x7f75d4072410 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75d27fc640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d4072410 0x7f75d4158990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f75c800c750 tx=0x7f75c800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75bb7fe640 1 -- 172.21.15.186:0/707319594 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8018070 con 0x7f75d4072410 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75bb7fe640 1 -- 172.21.15.186:0/707319594 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c80072d0 con 0x7f75d4072410 2024-03-28T15:03:37.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75d93b5640 1 -- 172.21.15.186:0/707319594 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d41548c0 con 0x7f75d4072410 2024-03-28T15:03:37.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75bb7fe640 1 -- 172.21.15.186:0/707319594 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c80047e0 con 0x7f75d4072410 2024-03-28T15:03:37.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.223+0000 7f75d93b5640 1 -- 172.21.15.186:0/707319594 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d4154e80 con 0x7f75d4072410 2024-03-28T15:03:37.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.224+0000 7f75d93b5640 1 -- 172.21.15.186:0/707319594 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75d4069500 con 0x7f75d4072410 2024-03-28T15:03:37.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.225+0000 7f75bb7fe640 1 -- 172.21.15.186:0/707319594 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f75c801b070 con 0x7f75d4072410 2024-03-28T15:03:37.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.225+0000 7f75bb7fe640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f75c006c560 0x7f75c006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:37.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.225+0000 7f75bb7fe640 1 -- 172.21.15.186:0/707319594 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5939+0+0 (secure 0 0 0) 0x7f75c808a960 con 0x7f75d4072410 2024-03-28T15:03:37.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.225+0000 7f75d2ffd640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f75c006c560 0x7f75c006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:37.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.226+0000 7f75d2ffd640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f75c006c560 0x7f75c006ea20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f75c4003aa0 tx=0x7f75c4009830 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:37.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.230+0000 7f75bb7fe640 1 -- 172.21.15.186:0/707319594 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f75d4069500 con 0x7f75d4072410 2024-03-28T15:03:37.443 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.442+0000 7f75d93b5640 1 -- 172.21.15.186:0/707319594 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75d40686d0 con 0x7f75d4072410 2024-03-28T15:03:37.444 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.442+0000 7f75bb7fe640 1 -- 172.21.15.186:0/707319594 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6844 (secure 0 0 0) 0x7f75d406eaf0 con 0x7f75d4072410 2024-03-28T15:03:37.444 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:37.444 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:03:37.444 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 13 2024-03-28T15:03:37.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.444+0000 7f75d93b5640 1 -- 172.21.15.186:0/707319594 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f75c006c560 msgr2=0x7f75c006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:37.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.444+0000 7f75d93b5640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f75c006c560 0x7f75c006ea20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f75c4003aa0 tx=0x7f75c4009830 comp rx=0 tx=0).stop 2024-03-28T15:03:37.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.444+0000 7f75d93b5640 1 -- 172.21.15.186:0/707319594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d4072410 msgr2=0x7f75d4158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:37.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.444+0000 7f75d93b5640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d4072410 0x7f75d4158990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f75c800c750 tx=0x7f75c800cc10 comp rx=0 tx=0).stop 2024-03-28T15:03:37.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.444+0000 7f75d93b5640 1 -- 172.21.15.186:0/707319594 shutdown_connections 2024-03-28T15:03:37.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.444+0000 7f75d93b5640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f75c006c560 0x7f75c006ea20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:37.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.445+0000 7f75d93b5640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75d41571a0 0x7f75d4158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.445+0000 7f75d93b5640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d4072410 0x7f75d4158990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.445+0000 7f75d93b5640 1 --2- 172.21.15.186:0/707319594 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75d406f410 0x7f75d40cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.445+0000 7f75d93b5640 1 -- 172.21.15.186:0/707319594 >> 172.21.15.186:0/707319594 conn(0x7f75d40cd360 msgr2=0x7f75d40c9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.445+0000 7f75d93b5640 1 -- 172.21.15.186:0/707319594 shutdown_connections 2024-03-28T15:03:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:37.445+0000 7f75d93b5640 1 -- 172.21.15.186:0/707319594 wait complete. 2024-03-28T15:03:37.456 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:03:39.457 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:39.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.570+0000 7f97f9c0e640 1 Processor -- start 2024-03-28T15:03:39.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.570+0000 7f97f9c0e640 1 -- start start 2024-03-28T15:03:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.571+0000 7f97f9c0e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f97f4152810 0x7f97f4158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.571+0000 7f97f9c0e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f97f4071340 0x7f97f406c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.571+0000 7f97f9c0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f4072df0 con 0x7f97f406ce60 2024-03-28T15:03:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.571+0000 7f97f9c0e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f97f4159500 con 0x7f97f4071340 2024-03-28T15:03:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.571+0000 7f97f9c0e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f97f40c8660 con 0x7f97f4152810 2024-03-28T15:03:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.571+0000 7f97eaffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f97f4071340 0x7f97f406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:39.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.571+0000 7f97f3fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f97f406ce60 0x7f97f40c6cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:43830/0 (socket says 172.21.15.186:43830) 2024-03-28T15:03:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.571+0000 7f97f3fff640 1 -- 172.21.15.186:0/837654222 learned_addr learned my addr 172.21.15.186:0/837654222 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:03:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.571+0000 7f97f37fe640 1 --2- 172.21.15.186:0/837654222 >> v2:172.21.15.191:3300/0 conn(0x7f97f4152810 0x7f97f4158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.572+0000 7f97eaffd640 1 -- 172.21.15.186:0/837654222 >> v2:172.21.15.191:3300/0 conn(0x7f97f4152810 msgr2=0x7f97f4158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.572+0000 7f97eaffd640 1 --2- 172.21.15.186:0/837654222 >> v2:172.21.15.191:3300/0 conn(0x7f97f4152810 0x7f97f4158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.572+0000 7f97eaffd640 1 -- 172.21.15.186:0/837654222 >> v1:172.21.15.186:6789/0 conn(0x7f97f406ce60 legacy=0x7f97f40c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.572+0000 7f97eaffd640 1 -- 172.21.15.186:0/837654222 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f4071d50 con 0x7f97f4071340 2024-03-28T15:03:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.572+0000 7f97f2ffd640 1 -- 172.21.15.186:0/837654222 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4002390333 0 0) 0x7f97f4072df0 con 0x7f97f406ce60 2024-03-28T15:03:39.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.572+0000 7f97eaffd640 1 --2- 172.21.15.186:0/837654222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97f4071340 0x7f97f406c920 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0097a0 tx=0x7f97dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04cbd499b0e055d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:39.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.572+0000 7f97f2ffd640 1 -- 172.21.15.186:0/837654222 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc035020 con 0x7f97f4071340 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.572+0000 7f97f2ffd640 1 -- 172.21.15.186:0/837654222 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97dc031640 con 0x7f97f4071340 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.572+0000 7f97f2ffd640 1 -- 172.21.15.186:0/837654222 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc02d040 con 0x7f97f4071340 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.573+0000 7f97f9c0e640 1 -- 172.21.15.186:0/837654222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97f4071340 msgr2=0x7f97f406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.573+0000 7f97f9c0e640 1 --2- 172.21.15.186:0/837654222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97f4071340 0x7f97f406c920 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0097a0 tx=0x7f97dc02b8f0 comp rx=0 tx=0).stop 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.573+0000 7f97f9c0e640 1 -- 172.21.15.186:0/837654222 shutdown_connections 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.573+0000 7f97f9c0e640 1 --2- 172.21.15.186:0/837654222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97f4071340 0x7f97f406c920 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.573+0000 7f97f9c0e640 1 --2- 172.21.15.186:0/837654222 >> v2:172.21.15.191:3300/0 conn(0x7f97f4152810 0x7f97f4158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.573+0000 7f97f9c0e640 1 -- 172.21.15.186:0/837654222 >> 172.21.15.186:0/837654222 conn(0x7f97f40c8a20 msgr2=0x7f97f40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.573+0000 7f97f9c0e640 1 -- 172.21.15.186:0/837654222 shutdown_connections 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.573+0000 7f97f9c0e640 1 -- 172.21.15.186:0/837654222 wait complete. 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.574+0000 7f97f9c0e640 1 Processor -- start 2024-03-28T15:03:39.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.574+0000 7f97f9c0e640 1 -- start start 2024-03-28T15:03:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97f9c0e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97f406ce60 0x7f97f414d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97f9c0e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97f4071340 0x7f97f4156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97f9c0e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97f4152810 0x7f97f41574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97f9c0e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97f40725f0 con 0x7f97f4071340 2024-03-28T15:03:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97f9c0e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f97f4072310 con 0x7f97f4152810 2024-03-28T15:03:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97f9c0e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f97f40728d0 con 0x7f97f406ce60 2024-03-28T15:03:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97eaffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97f4071340 0x7f97f4156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97eaffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97f4071340 0x7f97f4156fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52826/0 (socket says 172.21.15.186:52826) 2024-03-28T15:03:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97eaffd640 1 -- 172.21.15.186:0/1125954199 learned_addr learned my addr 172.21.15.186:0/1125954199 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97f3fff640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97f4152810 0x7f97f41574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:39.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97f37fe640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97f406ce60 0x7f97f414d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97eaffd640 1 -- 172.21.15.186:0/1125954199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97f406ce60 msgr2=0x7f97f414d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97eaffd640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97f406ce60 0x7f97f414d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97eaffd640 1 -- 172.21.15.186:0/1125954199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97f4152810 msgr2=0x7f97f41574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97eaffd640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97f4152810 0x7f97f41574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.575+0000 7f97eaffd640 1 -- 172.21.15.186:0/1125954199 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f4150d90 con 0x7f97f4071340 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.576+0000 7f97eaffd640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97f4071340 0x7f97f4156fa0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f97dc000c00 tx=0x7f97dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.576+0000 7f97f0ff9640 1 -- 172.21.15.186:0/1125954199 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc040070 con 0x7f97f4071340 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.576+0000 7f97f0ff9640 1 -- 172.21.15.186:0/1125954199 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97dc0399f0 con 0x7f97f4071340 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.576+0000 7f97f9c0e640 1 -- 172.21.15.186:0/1125954199 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f4156ac0 con 0x7f97f4071340 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.576+0000 7f97f0ff9640 1 -- 172.21.15.186:0/1125954199 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc035040 con 0x7f97f4071340 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.576+0000 7f97f9c0e640 1 -- 172.21.15.186:0/1125954199 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f41504f0 con 0x7f97f4071340 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.577+0000 7f97f9c0e640 1 -- 172.21.15.186:0/1125954199 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f4152bf0 con 0x7f97f4071340 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.577+0000 7f97f0ff9640 1 -- 172.21.15.186:0/1125954199 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f97dc04b060 con 0x7f97f4071340 2024-03-28T15:03:39.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.578+0000 7f97f0ff9640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97d406c530 0x7f97d406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:39.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.578+0000 7f97f0ff9640 1 -- 172.21.15.186:0/1125954199 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5939+0+0 (secure 0 0 0) 0x7f97dc0ad090 con 0x7f97f4071340 2024-03-28T15:03:39.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.578+0000 7f97f37fe640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97d406c530 0x7f97d406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:39.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.578+0000 7f97f37fe640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97d406c530 0x7f97d406e9f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009a30 tx=0x7f97e4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:39.582 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.582+0000 7f97f0ff9640 1 -- 172.21.15.186:0/1125954199 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f97f4152bf0 con 0x7f97f4071340 2024-03-28T15:03:39.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.796+0000 7f97f9c0e640 1 -- 172.21.15.186:0/1125954199 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97f4069500 con 0x7f97f4071340 2024-03-28T15:03:39.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.796+0000 7f97f0ff9640 1 -- 172.21.15.186:0/1125954199 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6844 (secure 0 0 0) 0x7f97dc07b5c0 con 0x7f97f4071340 2024-03-28T15:03:39.797 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:39.797 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:03:39.797 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 13 2024-03-28T15:03:39.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.798+0000 7f97f9c0e640 1 -- 172.21.15.186:0/1125954199 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97d406c530 msgr2=0x7f97d406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:39.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.798+0000 7f97f9c0e640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97d406c530 0x7f97d406e9f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009a30 tx=0x7f97e4008040 comp rx=0 tx=0).stop 2024-03-28T15:03:39.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.799+0000 7f97f9c0e640 1 -- 172.21.15.186:0/1125954199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97f4071340 msgr2=0x7f97f4156fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:39.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.799+0000 7f97f9c0e640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97f4071340 0x7f97f4156fa0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f97dc000c00 tx=0x7f97dc0046c0 comp rx=0 tx=0).stop 2024-03-28T15:03:39.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.799+0000 7f97f9c0e640 1 -- 172.21.15.186:0/1125954199 shutdown_connections 2024-03-28T15:03:39.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.799+0000 7f97f9c0e640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97d406c530 0x7f97d406e9f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:39.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.799+0000 7f97f9c0e640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97f4152810 0x7f97f41574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:39.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.799+0000 7f97f9c0e640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97f4071340 0x7f97f4156fa0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:39.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.799+0000 7f97f9c0e640 1 --2- 172.21.15.186:0/1125954199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97f406ce60 0x7f97f414d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:39.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.799+0000 7f97f9c0e640 1 -- 172.21.15.186:0/1125954199 >> 172.21.15.186:0/1125954199 conn(0x7f97f40c8a20 msgr2=0x7f97f40c6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:39.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.799+0000 7f97f9c0e640 1 -- 172.21.15.186:0/1125954199 shutdown_connections 2024-03-28T15:03:39.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:39.799+0000 7f97f9c0e640 1 -- 172.21.15.186:0/1125954199 wait complete. 2024-03-28T15:03:39.810 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:03:41.811 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:41.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.923+0000 7f676a89d640 1 Processor -- start 2024-03-28T15:03:41.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.923+0000 7f676a89d640 1 -- start start 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f676a89d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67641512b0 0x7f67640c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f676a89d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67640733f0 0x7f67640b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f676a89d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67640c1b40 con 0x7f6764150ed0 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f676a89d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f67640c1e20 con 0x7f67641512b0 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f676a89d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f67640c18a0 con 0x7f67640733f0 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f67637fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67641512b0 0x7f67640c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f67637fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67641512b0 0x7f67640c0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52842/0 (socket says 172.21.15.186:52842) 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f67637fe640 1 -- 172.21.15.186:0/1162585755 learned_addr learned my addr 172.21.15.186:0/1162585755 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f6768e3a640 1 --2- 172.21.15.186:0/1162585755 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67640733f0 0x7f67640b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f67637fe640 1 -- 172.21.15.186:0/1162585755 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67640733f0 msgr2=0x7f67640b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f67637fe640 1 --2- 172.21.15.186:0/1162585755 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67640733f0 0x7f67640b1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f67637fe640 1 -- 172.21.15.186:0/1162585755 >> v1:172.21.15.191:6789/0 conn(0x7f6764150ed0 legacy=0x7f6764156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:41.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.924+0000 7f67637fe640 1 -- 172.21.15.186:0/1162585755 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67640c15c0 con 0x7f67641512b0 2024-03-28T15:03:41.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.925+0000 7f67637fe640 1 --2- 172.21.15.186:0/1162585755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67641512b0 0x7f67640c0f40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6758009870 tx=0x7f675802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3886758b25051a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:41.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.925+0000 7f6762ffd640 1 -- 172.21.15.186:0/1162585755 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675802f840 con 0x7f67641512b0 2024-03-28T15:03:41.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.925+0000 7f6762ffd640 1 -- 172.21.15.186:0/1162585755 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f675802f9e0 con 0x7f67641512b0 2024-03-28T15:03:41.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.925+0000 7f6762ffd640 1 -- 172.21.15.186:0/1162585755 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675802fdb0 con 0x7f67641512b0 2024-03-28T15:03:41.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.926+0000 7f676a89d640 1 -- 172.21.15.186:0/1162585755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67641512b0 msgr2=0x7f67640c0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:41.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.926+0000 7f676a89d640 1 --2- 172.21.15.186:0/1162585755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67641512b0 0x7f67640c0f40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6758009870 tx=0x7f675802ba20 comp rx=0 tx=0).stop 2024-03-28T15:03:41.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.926+0000 7f676a89d640 1 -- 172.21.15.186:0/1162585755 shutdown_connections 2024-03-28T15:03:41.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.926+0000 7f676a89d640 1 --2- 172.21.15.186:0/1162585755 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67640733f0 0x7f67640b1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:41.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.926+0000 7f676a89d640 1 --2- 172.21.15.186:0/1162585755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67641512b0 0x7f67640c0f40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:41.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.926+0000 7f676a89d640 1 -- 172.21.15.186:0/1162585755 >> 172.21.15.186:0/1162585755 conn(0x7f67640c8a20 msgr2=0x7f67640c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:41.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.926+0000 7f676a89d640 1 -- 172.21.15.186:0/1162585755 shutdown_connections 2024-03-28T15:03:41.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.926+0000 7f676a89d640 1 -- 172.21.15.186:0/1162585755 wait complete. 2024-03-28T15:03:41.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.927+0000 7f676a89d640 1 Processor -- start 2024-03-28T15:03:41.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.927+0000 7f676a89d640 1 -- start start 2024-03-28T15:03:41.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f676a89d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67640733f0 0x7f67641eec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:41.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f676a89d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6764150ed0 0x7f67641ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:41.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f676a89d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67641512b0 0x7f67641f1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:41.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f676a89d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f676406e5b0 con 0x7f6764150ed0 2024-03-28T15:03:41.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f676a89d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6764072f60 con 0x7f67641512b0 2024-03-28T15:03:41.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f676a89d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f67640712b0 con 0x7f67640733f0 2024-03-28T15:03:41.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f67637fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6764150ed0 0x7f67641ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:41.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f67637fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6764150ed0 0x7f67641ee2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52850/0 (socket says 172.21.15.186:52850) 2024-03-28T15:03:41.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f67637fe640 1 -- 172.21.15.186:0/971734278 learned_addr learned my addr 172.21.15.186:0/971734278 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:41.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f6768e3a640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67641512b0 0x7f67641f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:41.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f6763fff640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67640733f0 0x7f67641eec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:41.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f67637fe640 1 -- 172.21.15.186:0/971734278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67640733f0 msgr2=0x7f67641eec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:41.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f67637fe640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67640733f0 0x7f67641eec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:41.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f67637fe640 1 -- 172.21.15.186:0/971734278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67641512b0 msgr2=0x7f67641f1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:41.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f67637fe640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67641512b0 0x7f67641f1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:41.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f67637fe640 1 -- 172.21.15.186:0/971734278 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67641f1c60 con 0x7f6764150ed0 2024-03-28T15:03:41.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f6768e3a640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67641512b0 0x7f67641f1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:03:41.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.928+0000 7f6763fff640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67640733f0 0x7f67641eec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:03:41.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.929+0000 7f67637fe640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6764150ed0 0x7f67641ee2b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6758000c00 tx=0x7f67580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:41.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.929+0000 7f6760ff9640 1 -- 172.21.15.186:0/971734278 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6758045070 con 0x7f6764150ed0 2024-03-28T15:03:41.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.929+0000 7f6760ff9640 1 -- 172.21.15.186:0/971734278 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6758037cb0 con 0x7f6764150ed0 2024-03-28T15:03:41.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.929+0000 7f676a89d640 1 -- 172.21.15.186:0/971734278 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67641ed7f0 con 0x7f6764150ed0 2024-03-28T15:03:41.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.929+0000 7f6760ff9640 1 -- 172.21.15.186:0/971734278 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6758037e60 con 0x7f6764150ed0 2024-03-28T15:03:41.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.929+0000 7f676a89d640 1 -- 172.21.15.186:0/971734278 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f676414a760 con 0x7f6764150ed0 2024-03-28T15:03:41.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.930+0000 7f676a89d640 1 -- 172.21.15.186:0/971734278 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6764069500 con 0x7f6764150ed0 2024-03-28T15:03:41.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.930+0000 7f6760ff9640 1 -- 172.21.15.186:0/971734278 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6758048070 con 0x7f6764150ed0 2024-03-28T15:03:41.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.931+0000 7f6760ff9640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f675006c530 0x7f675006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:41.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.931+0000 7f6760ff9640 1 -- 172.21.15.186:0/971734278 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5939+0+0 (secure 0 0 0) 0x7f67580afde0 con 0x7f6764150ed0 2024-03-28T15:03:41.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.931+0000 7f6763fff640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f675006c530 0x7f675006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:41.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.931+0000 7f6763fff640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f675006c530 0x7f675006e9f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6754001700 tx=0x7f6754009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:41.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:41.935+0000 7f6760ff9640 1 -- 172.21.15.186:0/971734278 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6758033030 con 0x7f6764150ed0 2024-03-28T15:03:42.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.150+0000 7f676a89d640 1 -- 172.21.15.186:0/971734278 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67640c81e0 con 0x7f6764150ed0 2024-03-28T15:03:42.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.151+0000 7f6760ff9640 1 -- 172.21.15.186:0/971734278 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6844 (secure 0 0 0) 0x7f675807e3a0 con 0x7f6764150ed0 2024-03-28T15:03:42.151 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:42.151 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:03:42.152 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 13 2024-03-28T15:03:42.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 -- 172.21.15.186:0/971734278 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f675006c530 msgr2=0x7f675006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:42.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f675006c530 0x7f675006e9f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6754001700 tx=0x7f6754009040 comp rx=0 tx=0).stop 2024-03-28T15:03:42.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 -- 172.21.15.186:0/971734278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6764150ed0 msgr2=0x7f67641ee2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:42.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6764150ed0 0x7f67641ee2b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6758000c00 tx=0x7f67580046c0 comp rx=0 tx=0).stop 2024-03-28T15:03:42.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 -- 172.21.15.186:0/971734278 shutdown_connections 2024-03-28T15:03:42.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f675006c530 0x7f675006e9f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:42.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67641512b0 0x7f67641f1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:42.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6764150ed0 0x7f67641ee2b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:42.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 --2- 172.21.15.186:0/971734278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67640733f0 0x7f67641eec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:42.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 -- 172.21.15.186:0/971734278 >> 172.21.15.186:0/971734278 conn(0x7f67640c8a20 msgr2=0x7f67640c9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:42.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 -- 172.21.15.186:0/971734278 shutdown_connections 2024-03-28T15:03:42.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:42.153+0000 7f676a89d640 1 -- 172.21.15.186:0/971734278 wait complete. 2024-03-28T15:03:42.164 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:03:44.165 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:44.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.277+0000 7f8862b7a640 1 Processor -- start 2024-03-28T15:03:44.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.277+0000 7f8862b7a640 1 -- start start 2024-03-28T15:03:44.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.278+0000 7f8862b7a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f885c1512b0 0x7f885c0c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.278+0000 7f8862b7a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0733f0 0x7f885c0b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.278+0000 7f8862b7a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f885c0c1b40 con 0x7f885c150ed0 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.278+0000 7f8862b7a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f885c0c1e20 con 0x7f885c1512b0 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.278+0000 7f8862b7a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f885c0c18a0 con 0x7f885c0733f0 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.278+0000 7f8860916640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f885c150ed0 0x7f885c156d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:43846/0 (socket says 172.21.15.186:43846) 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.278+0000 7f8860916640 1 -- 172.21.15.186:0/3963547847 learned_addr learned my addr 172.21.15.186:0/3963547847 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.278+0000 7f8861117640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0733f0 0x7f885c0b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.278+0000 7f885bfff640 1 --2- 172.21.15.186:0/3963547847 >> v2:172.21.15.191:3300/0 conn(0x7f885c1512b0 0x7f885c0c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 551427669 0 0) 0x7f885c0c1b40 con 0x7f885c150ed0 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f884c003440 con 0x7f885c150ed0 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2487823810 0 0) 0x7f884c003440 con 0x7f885c150ed0 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0733f0 msgr2=0x7f885c0b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0733f0 0x7f885c0b1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:44.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 >> v2:172.21.15.191:3300/0 conn(0x7f885c1512b0 msgr2=0x7f885c0c0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 --2- 172.21.15.186:0/3963547847 >> v2:172.21.15.191:3300/0 conn(0x7f885c1512b0 0x7f885c0c0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885c0c15c0 con 0x7f885c150ed0 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f8850001670 con 0x7f885c150ed0 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 >> v1:172.21.15.186:6789/0 conn(0x7f885c150ed0 legacy=0x7f885c156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f884c003aa0 0x7f884c003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0733f0 0x7f884c00a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f884c00ae30 0x7f884c00d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f885c0c1e20 con 0x7f884c00ae30 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f885c0c18a0 con 0x7f884c003aa0 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f884c00dbe0 con 0x7f885c0733f0 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f8861117640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f884c00ae30 0x7f884c00d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f8860916640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f884c003aa0 0x7f884c003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:44.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f885bfff640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0733f0 0x7f884c00a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:44.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f8861117640 1 -- 172.21.15.186:0/3963547847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0733f0 msgr2=0x7f884c00a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:44.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f8861117640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0733f0 0x7f884c00a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:44.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f8861117640 1 -- 172.21.15.186:0/3963547847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f884c003aa0 msgr2=0x7f884c003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:44.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f8861117640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f884c003aa0 0x7f884c003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:44.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.279+0000 7f8861117640 1 -- 172.21.15.186:0/3963547847 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884c003440 con 0x7f884c00ae30 2024-03-28T15:03:44.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.280+0000 7f8861117640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f884c00ae30 0x7f884c00d2c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8854009fc0 tx=0x7f884c003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac198b4d51adaaa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:44.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.280+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854039070 con 0x7f884c00ae30 2024-03-28T15:03:44.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.280+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f885402cc20 con 0x7f884c00ae30 2024-03-28T15:03:44.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.280+0000 7f885b7fe640 1 -- 172.21.15.186:0/3963547847 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854009c10 con 0x7f884c00ae30 2024-03-28T15:03:44.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.281+0000 7f8862b7a640 1 -- 172.21.15.186:0/3963547847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f884c00ae30 msgr2=0x7f884c00d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:44.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.281+0000 7f8862b7a640 1 --2- 172.21.15.186:0/3963547847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f884c00ae30 0x7f884c00d2c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8854009fc0 tx=0x7f884c003730 comp rx=0 tx=0).stop 2024-03-28T15:03:44.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.281+0000 7f8861117640 1 -- 172.21.15.186:0/3963547847 reap_dead start 2024-03-28T15:03:44.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.281+0000 7f8862b7a640 1 -- 172.21.15.186:0/3963547847 shutdown_connections 2024-03-28T15:03:44.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.281+0000 7f8862b7a640 1 -- 172.21.15.186:0/3963547847 >> 172.21.15.186:0/3963547847 conn(0x7f885c0c8a20 msgr2=0x7f885c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:44.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.281+0000 7f8862b7a640 1 -- 172.21.15.186:0/3963547847 shutdown_connections 2024-03-28T15:03:44.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.281+0000 7f8862b7a640 1 -- 172.21.15.186:0/3963547847 wait complete. 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.282+0000 7f8862b7a640 1 Processor -- start 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.282+0000 7f8862b7a640 1 -- start start 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8862b7a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f884c00ae30 0x7f885c0cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8862b7a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0c9d20 0x7f885c1f3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8862b7a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c1f0830 0x7f885c14fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8862b7a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f885c06e5b0 con 0x7f885c1f0830 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8862b7a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f885c072f60 con 0x7f884c00ae30 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8862b7a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f885c0712b0 con 0x7f885c0c9d20 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8861117640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c1f0830 0x7f885c14fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8861117640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c1f0830 0x7f885c14fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52854/0 (socket says 172.21.15.186:52854) 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8861117640 1 -- 172.21.15.186:0/3014925990 learned_addr learned my addr 172.21.15.186:0/3014925990 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:44.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8860916640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f884c00ae30 0x7f885c0cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f885bfff640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0c9d20 0x7f885c1f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8861117640 1 -- 172.21.15.186:0/3014925990 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0c9d20 msgr2=0x7f885c1f3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8861117640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0c9d20 0x7f885c1f3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8861117640 1 -- 172.21.15.186:0/3014925990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f884c00ae30 msgr2=0x7f885c0cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8861117640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f884c00ae30 0x7f885c0cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8861117640 1 -- 172.21.15.186:0/3014925990 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885c1f3a40 con 0x7f885c1f0830 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f8860916640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f884c00ae30 0x7f885c0cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.283+0000 7f885bfff640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0c9d20 0x7f885c1f3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.284+0000 7f8861117640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c1f0830 0x7f885c14fe80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8854002230 tx=0x7f8854009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.284+0000 7f88597fa640 1 -- 172.21.15.186:0/3014925990 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854039030 con 0x7f885c1f0830 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.284+0000 7f88597fa640 1 -- 172.21.15.186:0/3014925990 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8854043070 con 0x7f885c1f0830 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.284+0000 7f88597fa640 1 -- 172.21.15.186:0/3014925990 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885403d7f0 con 0x7f885c1f0830 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.284+0000 7f8862b7a640 1 -- 172.21.15.186:0/3014925990 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885c1f3d20 con 0x7f885c1f0830 2024-03-28T15:03:44.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.284+0000 7f8862b7a640 1 -- 172.21.15.186:0/3014925990 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885c1f0030 con 0x7f885c1f0830 2024-03-28T15:03:44.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.285+0000 7f8862b7a640 1 -- 172.21.15.186:0/3014925990 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f885c1f4240 con 0x7f885c1f0830 2024-03-28T15:03:44.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.285+0000 7f88597fa640 1 -- 172.21.15.186:0/3014925990 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8854005d10 con 0x7f885c1f0830 2024-03-28T15:03:44.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.286+0000 7f88597fa640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f884006c560 0x7f884006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:44.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.286+0000 7f88597fa640 1 -- 172.21.15.186:0/3014925990 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5939+0+0 (secure 0 0 0) 0x7f88540adb10 con 0x7f885c1f0830 2024-03-28T15:03:44.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.286+0000 7f8860916640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f884006c560 0x7f884006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:44.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.286+0000 7f8860916640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f884006c560 0x7f884006ea20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8850000c00 tx=0x7f8850009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:44.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.290+0000 7f88597fa640 1 -- 172.21.15.186:0/3014925990 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f885c1f4240 con 0x7f885c1f0830 2024-03-28T15:03:44.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.507+0000 7f8862b7a640 1 -- 172.21.15.186:0/3014925990 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f885c1efe20 con 0x7f885c1f0830 2024-03-28T15:03:44.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.507+0000 7f88597fa640 1 -- 172.21.15.186:0/3014925990 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6844 (secure 0 0 0) 0x7f885407bfb0 con 0x7f885c1f0830 2024-03-28T15:03:44.508 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:44.508 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:02:26.265214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6834/185536353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":185536353},{"type":"v1","addr":"172.21.15.191:6834","nonce":185536353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-28T15:03:44.508 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 13 2024-03-28T15:03:44.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 -- 172.21.15.186:0/3014925990 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f884006c560 msgr2=0x7f884006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:44.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f884006c560 0x7f884006ea20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8850000c00 tx=0x7f8850009040 comp rx=0 tx=0).stop 2024-03-28T15:03:44.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 -- 172.21.15.186:0/3014925990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c1f0830 msgr2=0x7f885c14fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:44.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c1f0830 0x7f885c14fe80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8854002230 tx=0x7f8854009750 comp rx=0 tx=0).stop 2024-03-28T15:03:44.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 -- 172.21.15.186:0/3014925990 shutdown_connections 2024-03-28T15:03:44.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f884006c560 0x7f884006ea20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:44.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c1f0830 0x7f885c14fe80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:44.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f885c0c9d20 0x7f885c1f3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:44.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 --2- 172.21.15.186:0/3014925990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f884c00ae30 0x7f885c0cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:44.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 -- 172.21.15.186:0/3014925990 >> 172.21.15.186:0/3014925990 conn(0x7f885c0c8a20 msgr2=0x7f885c14e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:44.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 -- 172.21.15.186:0/3014925990 shutdown_connections 2024-03-28T15:03:44.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:44.510+0000 7f8862b7a640 1 -- 172.21.15.186:0/3014925990 wait complete. 2024-03-28T15:03:44.521 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:03:46.522 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:46.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.630+0000 7fe30d081640 1 Processor -- start 2024-03-28T15:03:46.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.630+0000 7fe30d081640 1 -- start start 2024-03-28T15:03:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.631+0000 7fe30d081640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe308154bd0 0x7fe30815a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.631+0000 7fe30d081640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe3080c57e0 0x7fe3080cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.631+0000 7fe30d081640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3080c6e00 con 0x7fe3080c53c0 2024-03-28T15:03:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.631+0000 7fe30d081640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fe3080c5ca0 con 0x7fe3080c57e0 2024-03-28T15:03:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.631+0000 7fe30d081640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fe30815b080 con 0x7fe308154bd0 2024-03-28T15:03:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.631+0000 7fe30759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe3080c57e0 0x7fe3080cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.631+0000 7fe30659c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe3080c53c0 0x7fe3080c66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:43856/0 (socket says 172.21.15.186:43856) 2024-03-28T15:03:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.631+0000 7fe30659c640 1 -- 172.21.15.186:0/369210681 learned_addr learned my addr 172.21.15.186:0/369210681 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:03:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.631+0000 7fe306d9d640 1 --2- 172.21.15.186:0/369210681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe308154bd0 0x7fe30815a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.632+0000 7fe30759e640 1 -- 172.21.15.186:0/369210681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe308154bd0 msgr2=0x7fe30815a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.632+0000 7fe30759e640 1 --2- 172.21.15.186:0/369210681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe308154bd0 0x7fe30815a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.632+0000 7fe30759e640 1 -- 172.21.15.186:0/369210681 >> v1:172.21.15.186:6789/0 conn(0x7fe3080c53c0 legacy=0x7fe3080c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:46.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.632+0000 7fe30759e640 1 -- 172.21.15.186:0/369210681 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30815b360 con 0x7fe3080c57e0 2024-03-28T15:03:46.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.632+0000 7fe305d9b640 1 -- 172.21.15.186:0/369210681 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1526086527 0 0) 0x7fe3080c6e00 con 0x7fe3080c53c0 2024-03-28T15:03:46.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.632+0000 7fe30759e640 1 --2- 172.21.15.186:0/369210681 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3080c57e0 0x7fe3080cb040 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe3000097a0 tx=0x7fe30002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1998cfa2986c8861 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:46.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.632+0000 7fe305d9b640 1 -- 172.21.15.186:0/369210681 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe300034070 con 0x7fe3080c57e0 2024-03-28T15:03:46.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.632+0000 7fe305d9b640 1 -- 172.21.15.186:0/369210681 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe300030610 con 0x7fe3080c57e0 2024-03-28T15:03:46.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.632+0000 7fe305d9b640 1 -- 172.21.15.186:0/369210681 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3000309e0 con 0x7fe3080c57e0 2024-03-28T15:03:46.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.633+0000 7fe30d081640 1 -- 172.21.15.186:0/369210681 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3080c57e0 msgr2=0x7fe3080cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:46.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.633+0000 7fe30d081640 1 --2- 172.21.15.186:0/369210681 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3080c57e0 0x7fe3080cb040 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe3000097a0 tx=0x7fe30002b960 comp rx=0 tx=0).stop 2024-03-28T15:03:46.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.633+0000 7fe30d081640 1 -- 172.21.15.186:0/369210681 shutdown_connections 2024-03-28T15:03:46.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.633+0000 7fe30d081640 1 --2- 172.21.15.186:0/369210681 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3080c57e0 0x7fe3080cb040 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.633+0000 7fe30d081640 1 --2- 172.21.15.186:0/369210681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe308154bd0 0x7fe30815a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.633+0000 7fe30d081640 1 -- 172.21.15.186:0/369210681 >> 172.21.15.186:0/369210681 conn(0x7fe3080731f0 msgr2=0x7fe308073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:46.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.633+0000 7fe30d081640 1 -- 172.21.15.186:0/369210681 shutdown_connections 2024-03-28T15:03:46.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.633+0000 7fe30d081640 1 -- 172.21.15.186:0/369210681 wait complete. 2024-03-28T15:03:46.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.634+0000 7fe30d081640 1 Processor -- start 2024-03-28T15:03:46.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe30d081640 1 -- start start 2024-03-28T15:03:46.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe30d081640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3080c53c0 0x7fe3081f0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe30d081640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3080c57e0 0x7fe3081f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe30d081640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe308154bd0 0x7fe30815d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe30d081640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe3080c61d0 con 0x7fe3080c57e0 2024-03-28T15:03:46.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe30d081640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe3080c5ef0 con 0x7fe308154bd0 2024-03-28T15:03:46.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe30d081640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fe3080cc390 con 0x7fe3080c53c0 2024-03-28T15:03:46.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe30659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3080c57e0 0x7fe3081f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe30659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3080c57e0 0x7fe3081f4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52868/0 (socket says 172.21.15.186:52868) 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe30659c640 1 -- 172.21.15.186:0/2236004436 learned_addr learned my addr 172.21.15.186:0/2236004436 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe30759e640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe308154bd0 0x7fe30815d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.635+0000 7fe306d9d640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3080c53c0 0x7fe3081f0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.636+0000 7fe30659c640 1 -- 172.21.15.186:0/2236004436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3080c53c0 msgr2=0x7fe3081f0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.636+0000 7fe30659c640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3080c53c0 0x7fe3081f0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.636+0000 7fe30659c640 1 -- 172.21.15.186:0/2236004436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe308154bd0 msgr2=0x7fe30815d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.636+0000 7fe30659c640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe308154bd0 0x7fe30815d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.636+0000 7fe30659c640 1 -- 172.21.15.186:0/2236004436 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3080b2af0 con 0x7fe3080c57e0 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.636+0000 7fe30659c640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3080c57e0 0x7fe3081f4050 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00c500 tx=0x7fe2fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.636+0000 7fe2ef7fe640 1 -- 172.21.15.186:0/2236004436 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc018070 con 0x7fe3080c57e0 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.636+0000 7fe2ef7fe640 1 -- 172.21.15.186:0/2236004436 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2fc007580 con 0x7fe3080c57e0 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.636+0000 7fe30d081640 1 -- 172.21.15.186:0/2236004436 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3081f3590 con 0x7fe3080c57e0 2024-03-28T15:03:46.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.636+0000 7fe2ef7fe640 1 -- 172.21.15.186:0/2236004436 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc004a20 con 0x7fe3080c57e0 2024-03-28T15:03:46.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.636+0000 7fe30d081640 1 -- 172.21.15.186:0/2236004436 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3080b3040 con 0x7fe3080c57e0 2024-03-28T15:03:46.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.637+0000 7fe30d081640 1 -- 172.21.15.186:0/2236004436 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe308069500 con 0x7fe3080c57e0 2024-03-28T15:03:46.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.637+0000 7fe2ef7fe640 1 -- 172.21.15.186:0/2236004436 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe2fc01b070 con 0x7fe3080c57e0 2024-03-28T15:03:46.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.638+0000 7fe2ef7fe640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe2f406c560 0x7fe2f406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.638+0000 7fe2ef7fe640 1 -- 172.21.15.186:0/2236004436 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7fe2fc08aa30 con 0x7fe3080c57e0 2024-03-28T15:03:46.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.638+0000 7fe306d9d640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe2f406c560 0x7fe2f406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.638+0000 7fe306d9d640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe2f406c560 0x7fe2f406ea20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8011340 tx=0x7fe2f800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:46.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.642+0000 7fe2ef7fe640 1 -- 172.21.15.186:0/2236004436 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fe2fc058970 con 0x7fe3080c57e0 2024-03-28T15:03:46.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.854+0000 7fe30d081640 1 -- 172.21.15.186:0/2236004436 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3080cbb60 con 0x7fe3080c57e0 2024-03-28T15:03:46.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.855+0000 7fe2ef7fe640 1 -- 172.21.15.186:0/2236004436 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6012 (secure 0 0 0) 0x7fe2fc05cc70 con 0x7fe3080c57e0 2024-03-28T15:03:46.855 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:46.855 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:03:46.183154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:03:46.856 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 14 2024-03-28T15:03:46.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 -- 172.21.15.186:0/2236004436 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe2f406c560 msgr2=0x7fe2f406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:46.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe2f406c560 0x7fe2f406ea20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8011340 tx=0x7fe2f800b040 comp rx=0 tx=0).stop 2024-03-28T15:03:46.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 -- 172.21.15.186:0/2236004436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3080c57e0 msgr2=0x7fe3081f4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:46.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3080c57e0 0x7fe3081f4050 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00c500 tx=0x7fe2fc00c9c0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 -- 172.21.15.186:0/2236004436 shutdown_connections 2024-03-28T15:03:46.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe2f406c560 0x7fe2f406ea20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe308154bd0 0x7fe30815d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3080c57e0 0x7fe3081f4050 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 --2- 172.21.15.186:0/2236004436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3080c53c0 0x7fe3081f0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 -- 172.21.15.186:0/2236004436 >> 172.21.15.186:0/2236004436 conn(0x7fe3080731f0 msgr2=0x7fe3080cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:46.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 -- 172.21.15.186:0/2236004436 shutdown_connections 2024-03-28T15:03:46.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.857+0000 7fe30d081640 1 -- 172.21.15.186:0/2236004436 wait complete. 2024-03-28T15:03:46.868 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-03-28T15:03:46.868 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-28T15:03:46.868 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:46.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.969+0000 7f52709d6640 1 Processor -- start 2024-03-28T15:03:46.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.969+0000 7f52709d6640 1 -- start start 2024-03-28T15:03:46.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.970+0000 7f52709d6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f526c0c53c0 0x7f526c0c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.970+0000 7f52709d6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f526c0c57e0 0x7f526c0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.970+0000 7f52709d6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f526c15b080 con 0x7f526c154bd0 2024-03-28T15:03:46.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.970+0000 7f52709d6640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f526c0c5ca0 con 0x7f526c0c57e0 2024-03-28T15:03:46.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.970+0000 7f52709d6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f526c15b360 con 0x7f526c0c53c0 2024-03-28T15:03:46.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.970+0000 7f526ad9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f526c0c57e0 0x7f526c0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.970+0000 7f526ad9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f526c0c57e0 0x7f526c0cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52878/0 (socket says 172.21.15.186:52878) 2024-03-28T15:03:46.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.970+0000 7f526a59c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f526c154bd0 0x7f526c15a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:43862/0 (socket says 172.21.15.186:43862) 2024-03-28T15:03:46.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.970+0000 7f526ad9d640 1 -- 172.21.15.186:0/3393068381 learned_addr learned my addr 172.21.15.186:0/3393068381 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:46.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.971+0000 7f5263fff640 1 --2- 172.21.15.186:0/3393068381 >> v2:172.21.15.191:3300/0 conn(0x7f526c0c53c0 0x7f526c0c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.971+0000 7f526ad9d640 1 -- 172.21.15.186:0/3393068381 >> v2:172.21.15.191:3300/0 conn(0x7f526c0c53c0 msgr2=0x7f526c0c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:46.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.971+0000 7f526ad9d640 1 --2- 172.21.15.186:0/3393068381 >> v2:172.21.15.191:3300/0 conn(0x7f526c0c53c0 0x7f526c0c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.971+0000 7f526ad9d640 1 -- 172.21.15.186:0/3393068381 >> v1:172.21.15.186:6789/0 conn(0x7f526c154bd0 legacy=0x7f526c15a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:46.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.971+0000 7f526ad9d640 1 -- 172.21.15.186:0/3393068381 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526c15b640 con 0x7f526c0c57e0 2024-03-28T15:03:46.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.971+0000 7f526ad9d640 1 --2- 172.21.15.186:0/3393068381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f526c0c57e0 0x7f526c0cb040 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f526c08d6b0 tx=0x7f526402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c83fc1b085dc28 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:46.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.972+0000 7f5269d9b640 1 -- 172.21.15.186:0/3393068381 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526402c960 con 0x7f526c0c57e0 2024-03-28T15:03:46.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.972+0000 7f5269d9b640 1 -- 172.21.15.186:0/3393068381 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f526402cb00 con 0x7f526c0c57e0 2024-03-28T15:03:46.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.972+0000 7f5269d9b640 1 -- 172.21.15.186:0/3393068381 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52640343f0 con 0x7f526c0c57e0 2024-03-28T15:03:46.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.972+0000 7f52709d6640 1 -- 172.21.15.186:0/3393068381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f526c0c57e0 msgr2=0x7f526c0cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:46.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.972+0000 7f52709d6640 1 --2- 172.21.15.186:0/3393068381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f526c0c57e0 0x7f526c0cb040 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f526c08d6b0 tx=0x7f526402be40 comp rx=0 tx=0).stop 2024-03-28T15:03:46.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.972+0000 7f52709d6640 1 -- 172.21.15.186:0/3393068381 shutdown_connections 2024-03-28T15:03:46.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.972+0000 7f52709d6640 1 --2- 172.21.15.186:0/3393068381 >> v2:172.21.15.191:3300/0 conn(0x7f526c0c53c0 0x7f526c0c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.972+0000 7f52709d6640 1 --2- 172.21.15.186:0/3393068381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f526c0c57e0 0x7f526c0cb040 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.972+0000 7f52709d6640 1 -- 172.21.15.186:0/3393068381 >> 172.21.15.186:0/3393068381 conn(0x7f526c0731f0 msgr2=0x7f526c073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:46.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.972+0000 7f52709d6640 1 -- 172.21.15.186:0/3393068381 shutdown_connections 2024-03-28T15:03:46.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.972+0000 7f52709d6640 1 -- 172.21.15.186:0/3393068381 wait complete. 2024-03-28T15:03:46.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.973+0000 7f52709d6640 1 Processor -- start 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f52709d6640 1 -- start start 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f52709d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f526c0c53c0 0x7f526c1f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f52709d6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f526c0c57e0 0x7f526c153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f52709d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f526c154bd0 0x7f526c159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f52709d6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f526c0cc390 con 0x7f526c0c57e0 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f52709d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f526c0c61d0 con 0x7f526c154bd0 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f52709d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f526c0c6c30 con 0x7f526c0c53c0 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f5263fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f526c0c57e0 0x7f526c153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f5263fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f526c0c57e0 0x7f526c153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52882/0 (socket says 172.21.15.186:52882) 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f5263fff640 1 -- 172.21.15.186:0/4270204228 learned_addr learned my addr 172.21.15.186:0/4270204228 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f526ad9d640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f526c154bd0 0x7f526c159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.974+0000 7f526a59c640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f526c0c53c0 0x7f526c1f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f5263fff640 1 -- 172.21.15.186:0/4270204228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f526c0c53c0 msgr2=0x7f526c1f0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f5263fff640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f526c0c53c0 0x7f526c1f0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f5263fff640 1 -- 172.21.15.186:0/4270204228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f526c154bd0 msgr2=0x7f526c159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f5263fff640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f526c154bd0 0x7f526c159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f5263fff640 1 -- 172.21.15.186:0/4270204228 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526c1e7090 con 0x7f526c0c57e0 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f526a59c640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f526c0c53c0 0x7f526c1f0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f526ad9d640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f526c154bd0 0x7f526c159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f5263fff640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f526c0c57e0 0x7f526c153ca0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f525400d620 tx=0x7f525400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f5262ffd640 1 -- 172.21.15.186:0/4270204228 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525400b620 con 0x7f526c0c57e0 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f5262ffd640 1 -- 172.21.15.186:0/4270204228 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5254019070 con 0x7f526c0c57e0 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f52709d6640 1 -- 172.21.15.186:0/4270204228 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526c1e7370 con 0x7f526c0c57e0 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f5262ffd640 1 -- 172.21.15.186:0/4270204228 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525400ba00 con 0x7f526c0c57e0 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.975+0000 7f52709d6640 1 -- 172.21.15.186:0/4270204228 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f526c1e7930 con 0x7f526c0c57e0 2024-03-28T15:03:46.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.976+0000 7f52709d6640 1 -- 172.21.15.186:0/4270204228 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f526c068ff0 con 0x7f526c0c57e0 2024-03-28T15:03:46.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.976+0000 7f5262ffd640 1 -- 172.21.15.186:0/4270204228 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f525400bba0 con 0x7f526c0c57e0 2024-03-28T15:03:46.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.977+0000 7f5262ffd640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f524c06c560 0x7f524c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:46.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.977+0000 7f5262ffd640 1 -- 172.21.15.186:0/4270204228 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7f525408bc70 con 0x7f526c0c57e0 2024-03-28T15:03:46.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.977+0000 7f526a59c640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f524c06c560 0x7f524c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:46.982 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.977+0000 7f526a59c640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f524c06c560 0x7f524c06ea20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f525c000f60 tx=0x7f525c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:46.982 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:46.981+0000 7f5262ffd640 1 -- 172.21.15.186:0/4270204228 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f526c068ff0 con 0x7f526c0c57e0 2024-03-28T15:03:47.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.196+0000 7f52709d6640 1 -- 172.21.15.186:0/4270204228 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f526c069500 con 0x7f526c0c57e0 2024-03-28T15:03:47.197 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.197+0000 7f5262ffd640 1 -- 172.21.15.186:0/4270204228 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6012 (secure 0 0 0) 0x7f525405a0e0 con 0x7f526c0c57e0 2024-03-28T15:03:47.197 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:47.197 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:03:46.183154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:03:47.198 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 14 2024-03-28T15:03:47.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 -- 172.21.15.186:0/4270204228 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f524c06c560 msgr2=0x7f524c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:47.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f524c06c560 0x7f524c06ea20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f525c000f60 tx=0x7f525c009040 comp rx=0 tx=0).stop 2024-03-28T15:03:47.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 -- 172.21.15.186:0/4270204228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f526c0c57e0 msgr2=0x7f526c153ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:47.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f526c0c57e0 0x7f526c153ca0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f525400d620 tx=0x7f525400dae0 comp rx=0 tx=0).stop 2024-03-28T15:03:47.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 -- 172.21.15.186:0/4270204228 shutdown_connections 2024-03-28T15:03:47.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f524c06c560 0x7f524c06ea20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:47.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f526c154bd0 0x7f526c159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:47.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f526c0c57e0 0x7f526c153ca0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:47.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 --2- 172.21.15.186:0/4270204228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f526c0c53c0 0x7f526c1f0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:47.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 -- 172.21.15.186:0/4270204228 >> 172.21.15.186:0/4270204228 conn(0x7f526c0731f0 msgr2=0x7f526c0cb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:47.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 -- 172.21.15.186:0/4270204228 shutdown_connections 2024-03-28T15:03:47.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:47.199+0000 7f52709d6640 1 -- 172.21.15.186:0/4270204228 wait complete. 2024-03-28T15:03:47.211 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 14, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T15:01:31.425299+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 14, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4879": { "addr": "172.21.15.191:6837/3828930138", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3828930138, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3828930138, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4879, "incarnation": 14, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T15:03:46.183154+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4879 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/1278716016", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6833", "nonce": 1278716016, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1278716016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6835/2068761001", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2068761001, "type": "v2" }, { "addr": "172.21.15.186:6835", "nonce": 2068761001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2344737463", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6837", "nonce": 2344737463, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2344737463, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4284, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/1548903381", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 1548903381, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 1548903381, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T15:03:49.212 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:49.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.326+0000 7fef868fb640 1 Processor -- start 2024-03-28T15:03:49.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.326+0000 7fef868fb640 1 -- start start 2024-03-28T15:03:49.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.326+0000 7fef868fb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef80152810 0x7fef80158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:49.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.326+0000 7fef868fb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef80072df0 con 0x7fef80071340 2024-03-28T15:03:49.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.326+0000 7fef868fb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef800c8660 con 0x7fef800c8090 2024-03-28T15:03:49.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.326+0000 7fef868fb640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fef80159500 con 0x7fef80152810 2024-03-28T15:03:49.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7f7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fef80071340 0x7fef8006c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:43670/0 (socket says 172.21.15.186:43670) 2024-03-28T15:03:49.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7f7fe640 1 -- 172.21.15.186:0/2785874361 learned_addr learned my addr 172.21.15.186:0/2785874361 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:03:49.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388309971 0 0) 0x7fef80072df0 con 0x7fef80071340 2024-03-28T15:03:49.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef68003520 con 0x7fef80071340 2024-03-28T15:03:49.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7ffff640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef80152810 0x7fef80158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:49.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 818469493 0 0) 0x7fef68003520 con 0x7fef80071340 2024-03-28T15:03:49.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef80152810 msgr2=0x7fef80158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef80152810 0x7fef80158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 >> v1:172.21.15.191:6789/0 conn(0x7fef800c8090 legacy=0x7fef800c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef80071d50 con 0x7fef80071340 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7fef74001670 con 0x7fef80071340 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 >> v1:172.21.15.186:6789/0 conn(0x7fef80071340 legacy=0x7fef8006c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef680039e0 0x7fef68003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef80152810 0x7fef68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 0x7fef680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef80159500 con 0x7fef68004d40 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fef6800ba70 con 0x7fef680039e0 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fef6800bc70 con 0x7fef80152810 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef84e98640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 0x7fef680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.327+0000 7fef7ffff640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef680039e0 0x7fef68003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:49.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.328+0000 7fef7f7fe640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef80152810 0x7fef68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.328+0000 7fef84e98640 1 -- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef80152810 msgr2=0x7fef68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.328+0000 7fef84e98640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef80152810 0x7fef68004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.328+0000 7fef84e98640 1 -- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef680039e0 msgr2=0x7fef68003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.328+0000 7fef84e98640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef680039e0 0x7fef68003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.328+0000 7fef84e98640 1 -- 172.21.15.186:0/2785874361 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef68003520 con 0x7fef68004d40 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.328+0000 7fef7f7fe640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef80152810 0x7fef68004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.328+0000 7fef84e98640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 0x7fef680051c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fef78001e20 tx=0x7fef68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986925a1dcaa770 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.329+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78001a70 con 0x7fef68004d40 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.329+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef78003b70 con 0x7fef68004d40 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.329+0000 7fef7effd640 1 -- 172.21.15.186:0/2785874361 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef780050d0 con 0x7fef68004d40 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.329+0000 7fef868fb640 1 -- 172.21.15.186:0/2785874361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 msgr2=0x7fef680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.329+0000 7fef868fb640 1 --2- 172.21.15.186:0/2785874361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 0x7fef680051c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fef78001e20 tx=0x7fef68005830 comp rx=0 tx=0).stop 2024-03-28T15:03:49.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.329+0000 7fef84e98640 1 -- 172.21.15.186:0/2785874361 reap_dead start 2024-03-28T15:03:49.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.329+0000 7fef868fb640 1 -- 172.21.15.186:0/2785874361 shutdown_connections 2024-03-28T15:03:49.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.329+0000 7fef868fb640 1 -- 172.21.15.186:0/2785874361 >> 172.21.15.186:0/2785874361 conn(0x7fef800c8a20 msgr2=0x7fef800c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:49.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.329+0000 7fef868fb640 1 -- 172.21.15.186:0/2785874361 shutdown_connections 2024-03-28T15:03:49.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.329+0000 7fef868fb640 1 -- 172.21.15.186:0/2785874361 wait complete. 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.330+0000 7fef868fb640 1 Processor -- start 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef868fb640 1 -- start start 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef868fb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 0x7fef801559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef868fb640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef801581c0 0x7fef801577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef868fb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef800ca760 0x7fef800c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef868fb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef800725f0 con 0x7fef68004d40 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef868fb640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fef80072310 con 0x7fef801581c0 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef868fb640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fef800728d0 con 0x7fef800ca760 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef7ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 0x7fef801559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef7ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 0x7fef801559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34368/0 (socket says 172.21.15.186:34368) 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef7ffff640 1 -- 172.21.15.186:0/1775171757 learned_addr learned my addr 172.21.15.186:0/1775171757 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:49.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef7f7fe640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef801581c0 0x7fef801577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.331+0000 7fef84e98640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef800ca760 0x7fef800c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef7ffff640 1 -- 172.21.15.186:0/1775171757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef800ca760 msgr2=0x7fef800c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef7ffff640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef800ca760 0x7fef800c1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef7ffff640 1 -- 172.21.15.186:0/1775171757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef801581c0 msgr2=0x7fef801577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef7ffff640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef801581c0 0x7fef801577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef7ffff640 1 -- 172.21.15.186:0/1775171757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef801f0d60 con 0x7fef68004d40 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef84e98640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef800ca760 0x7fef800c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef7ffff640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 0x7fef801559e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fef70011910 tx=0x7fef70011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef7cff9640 1 -- 172.21.15.186:0/1775171757 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7000faf0 con 0x7fef68004d40 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef7cff9640 1 -- 172.21.15.186:0/1775171757 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef7000fc90 con 0x7fef68004d40 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef868fb640 1 -- 172.21.15.186:0/1775171757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef801f1000 con 0x7fef68004d40 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef7cff9640 1 -- 172.21.15.186:0/1775171757 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70014440 con 0x7fef68004d40 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.332+0000 7fef868fb640 1 -- 172.21.15.186:0/1775171757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef800c13c0 con 0x7fef68004d40 2024-03-28T15:03:49.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.333+0000 7fef868fb640 1 -- 172.21.15.186:0/1775171757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef80069500 con 0x7fef68004d40 2024-03-28T15:03:49.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.333+0000 7fef7cff9640 1 -- 172.21.15.186:0/1775171757 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fef70012040 con 0x7fef68004d40 2024-03-28T15:03:49.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.334+0000 7fef7cff9640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef6c06c530 0x7fef6c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:49.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.334+0000 7fef7cff9640 1 -- 172.21.15.186:0/1775171757 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7fef7008bdc0 con 0x7fef68004d40 2024-03-28T15:03:49.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.334+0000 7fef7f7fe640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef6c06c530 0x7fef6c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:49.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.334+0000 7fef7f7fe640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef6c06c530 0x7fef6c06e9f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fef800b2010 tx=0x7fef74007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:49.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.338+0000 7fef7cff9640 1 -- 172.21.15.186:0/1775171757 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fef70019030 con 0x7fef68004d40 2024-03-28T15:03:49.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.555+0000 7fef868fb640 1 -- 172.21.15.186:0/1775171757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef800c77f0 con 0x7fef68004d40 2024-03-28T15:03:49.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.555+0000 7fef7cff9640 1 -- 172.21.15.186:0/1775171757 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6012 (secure 0 0 0) 0x7fef7005a2c0 con 0x7fef68004d40 2024-03-28T15:03:49.556 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:49.556 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:03:46.183154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:03:49.556 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 14 2024-03-28T15:03:49.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 -- 172.21.15.186:0/1775171757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef6c06c530 msgr2=0x7fef6c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:49.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef6c06c530 0x7fef6c06e9f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fef800b2010 tx=0x7fef74007040 comp rx=0 tx=0).stop 2024-03-28T15:03:49.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 -- 172.21.15.186:0/1775171757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 msgr2=0x7fef801559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:49.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 0x7fef801559e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fef70011910 tx=0x7fef70011dd0 comp rx=0 tx=0).stop 2024-03-28T15:03:49.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 -- 172.21.15.186:0/1775171757 shutdown_connections 2024-03-28T15:03:49.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fef6c06c530 0x7fef6c06e9f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:49.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef800ca760 0x7fef800c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:49.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef801581c0 0x7fef801577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:49.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 --2- 172.21.15.186:0/1775171757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef68004d40 0x7fef801559e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:49.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 -- 172.21.15.186:0/1775171757 >> 172.21.15.186:0/1775171757 conn(0x7fef800c8a20 msgr2=0x7fef800c4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:49.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 -- 172.21.15.186:0/1775171757 shutdown_connections 2024-03-28T15:03:49.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:49.558+0000 7fef868fb640 1 -- 172.21.15.186:0/1775171757 wait complete. 2024-03-28T15:03:49.569 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:03:51.570 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:51.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.683+0000 7f8e6bdc9640 1 Processor -- start 2024-03-28T15:03:51.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.684+0000 7f8e6bdc9640 1 -- start start 2024-03-28T15:03:51.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.684+0000 7f8e6bdc9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8e64154bd0 0x7f8e6415a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:51.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.684+0000 7f8e6bdc9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e640c6e00 con 0x7f8e640c57e0 2024-03-28T15:03:51.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.684+0000 7f8e6bdc9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6415b080 con 0x7f8e640c53c0 2024-03-28T15:03:51.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.684+0000 7f8e6bdc9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f8e640c5ca0 con 0x7f8e64154bd0 2024-03-28T15:03:51.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.685+0000 7f8e6a366640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8e640c57e0 0x7f8e640cb040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:43680/0 (socket says 172.21.15.186:43680) 2024-03-28T15:03:51.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.685+0000 7f8e6a366640 1 -- 172.21.15.186:0/1820288126 learned_addr learned my addr 172.21.15.186:0/1820288126 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:03:51.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.685+0000 7f8e69b65640 1 --2- 172.21.15.186:0/1820288126 >> v2:172.21.15.191:3300/0 conn(0x7f8e64154bd0 0x7f8e6415a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:51.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.685+0000 7f8e68b63640 1 -- 172.21.15.186:0/1820288126 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3389467701 0 0) 0x7f8e640c6e00 con 0x7f8e640c57e0 2024-03-28T15:03:51.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.685+0000 7f8e68b63640 1 -- 172.21.15.186:0/1820288126 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e50003520 con 0x7f8e640c57e0 2024-03-28T15:03:51.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.685+0000 7f8e68b63640 1 -- 172.21.15.186:0/1820288126 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2557862421 0 0) 0x7f8e6415b080 con 0x7f8e640c53c0 2024-03-28T15:03:51.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.685+0000 7f8e68b63640 1 -- 172.21.15.186:0/1820288126 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e640c6e00 con 0x7f8e640c53c0 2024-03-28T15:03:51.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.685+0000 7f8e69b65640 1 -- 172.21.15.186:0/1820288126 >> v1:172.21.15.191:6789/0 conn(0x7f8e640c53c0 legacy=0x7f8e640c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:51.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.685+0000 7f8e69b65640 1 -- 172.21.15.186:0/1820288126 >> v1:172.21.15.186:6789/0 conn(0x7f8e640c57e0 legacy=0x7f8e640cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:51.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.685+0000 7f8e69b65640 1 -- 172.21.15.186:0/1820288126 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6415b360 con 0x7f8e64154bd0 2024-03-28T15:03:51.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.686+0000 7f8e69b65640 1 --2- 172.21.15.186:0/1820288126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e64154bd0 0x7f8e6415a7d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8e540097a0 tx=0x7f8e5402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f75ba7fd679d81c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.686+0000 7f8e68b63640 1 -- 172.21.15.186:0/1820288126 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5402f810 con 0x7f8e64154bd0 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.686+0000 7f8e68b63640 1 -- 172.21.15.186:0/1820288126 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e5402f9b0 con 0x7f8e64154bd0 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.686+0000 7f8e68b63640 1 -- 172.21.15.186:0/1820288126 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5402fd80 con 0x7f8e64154bd0 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.686+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/1820288126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e64154bd0 msgr2=0x7f8e6415a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.686+0000 7f8e6bdc9640 1 --2- 172.21.15.186:0/1820288126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e64154bd0 0x7f8e6415a7d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8e540097a0 tx=0x7f8e5402b9f0 comp rx=0 tx=0).stop 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.686+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/1820288126 shutdown_connections 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.686+0000 7f8e6bdc9640 1 --2- 172.21.15.186:0/1820288126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e64154bd0 0x7f8e6415a7d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.686+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/1820288126 >> 172.21.15.186:0/1820288126 conn(0x7f8e640731f0 msgr2=0x7f8e64073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.687+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/1820288126 shutdown_connections 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.687+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/1820288126 wait complete. 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.688+0000 7f8e6bdc9640 1 Processor -- start 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.688+0000 7f8e6bdc9640 1 -- start start 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.688+0000 7f8e6bdc9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e640c53c0 0x7f8e641f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:51.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.688+0000 7f8e6bdc9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e640c57e0 0x7f8e6414a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.688+0000 7f8e6bdc9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e64154bd0 0x7f8e641594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.688+0000 7f8e6bdc9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e640c61d0 con 0x7f8e640c57e0 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.688+0000 7f8e6bdc9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8e640c5ef0 con 0x7f8e64154bd0 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.688+0000 7f8e6bdc9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8e640cc390 con 0x7f8e640c53c0 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.688+0000 7f8e69364640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e640c57e0 0x7f8e6414a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e6a366640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e64154bd0 0x7f8e641594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e69364640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e640c57e0 0x7f8e6414a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34372/0 (socket says 172.21.15.186:34372) 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e69364640 1 -- 172.21.15.186:0/3720730620 learned_addr learned my addr 172.21.15.186:0/3720730620 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e69b65640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e640c53c0 0x7f8e641f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e69364640 1 -- 172.21.15.186:0/3720730620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e640c53c0 msgr2=0x7f8e641f42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e69364640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e640c53c0 0x7f8e641f42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e69364640 1 -- 172.21.15.186:0/3720730620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e64154bd0 msgr2=0x7f8e641594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e69364640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e64154bd0 0x7f8e641594a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:51.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e69364640 1 -- 172.21.15.186:0/3720730620 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e64155cc0 con 0x7f8e640c57e0 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e69364640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e640c57e0 0x7f8e6414a370 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000c500 tx=0x7f8e6000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e5a7fc640 1 -- 172.21.15.186:0/3720730620 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e600052d0 con 0x7f8e640c57e0 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e5a7fc640 1 -- 172.21.15.186:0/3720730620 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e60005490 con 0x7f8e640c57e0 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/3720730620 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e64155fa0 con 0x7f8e640c57e0 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e5a7fc640 1 -- 172.21.15.186:0/3720730620 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60012870 con 0x7f8e640c57e0 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.689+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/3720730620 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e64156560 con 0x7f8e640c57e0 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.690+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/3720730620 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e64069500 con 0x7f8e640c57e0 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.691+0000 7f8e5a7fc640 1 -- 172.21.15.186:0/3720730620 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8e60012a10 con 0x7f8e640c57e0 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.691+0000 7f8e5a7fc640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8e4806c530 0x7f8e4806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.691+0000 7f8e5a7fc640 1 -- 172.21.15.186:0/3720730620 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7f8e6008b380 con 0x7f8e640c57e0 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.691+0000 7f8e69b65640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8e4806c530 0x7f8e4806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:51.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.692+0000 7f8e69b65640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8e4806c530 0x7f8e4806e9f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8e54036000 tx=0x7f8e54008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:51.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.696+0000 7f8e5a7fc640 1 -- 172.21.15.186:0/3720730620 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8e60059240 con 0x7f8e640c57e0 2024-03-28T15:03:51.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.907+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/3720730620 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e640cbb60 con 0x7f8e640c57e0 2024-03-28T15:03:51.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.907+0000 7f8e5a7fc640 1 -- 172.21.15.186:0/3720730620 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6012 (secure 0 0 0) 0x7f8e6005d540 con 0x7f8e640c57e0 2024-03-28T15:03:51.908 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:51.908 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:03:46.183154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:03:51.908 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 14 2024-03-28T15:03:51.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.909+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/3720730620 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8e4806c530 msgr2=0x7f8e4806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:51.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.909+0000 7f8e6bdc9640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8e4806c530 0x7f8e4806e9f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8e54036000 tx=0x7f8e54008030 comp rx=0 tx=0).stop 2024-03-28T15:03:51.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.909+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/3720730620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e640c57e0 msgr2=0x7f8e6414a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:51.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.909+0000 7f8e6bdc9640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e640c57e0 0x7f8e6414a370 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000c500 tx=0x7f8e6000c9c0 comp rx=0 tx=0).stop 2024-03-28T15:03:51.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.909+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/3720730620 shutdown_connections 2024-03-28T15:03:51.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.909+0000 7f8e6bdc9640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8e4806c530 0x7f8e4806e9f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:51.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.909+0000 7f8e6bdc9640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e64154bd0 0x7f8e641594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:51.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.909+0000 7f8e6bdc9640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e640c57e0 0x7f8e6414a370 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:51.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.909+0000 7f8e6bdc9640 1 --2- 172.21.15.186:0/3720730620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e640c53c0 0x7f8e641f42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:51.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.910+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/3720730620 >> 172.21.15.186:0/3720730620 conn(0x7f8e640731f0 msgr2=0x7f8e640cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:51.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.910+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/3720730620 shutdown_connections 2024-03-28T15:03:51.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:51.910+0000 7f8e6bdc9640 1 -- 172.21.15.186:0/3720730620 wait complete. 2024-03-28T15:03:51.921 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:03:53.922 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:54.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.037+0000 7effdd2c9640 1 Processor -- start 2024-03-28T15:03:54.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.037+0000 7effdd2c9640 1 -- start start 2024-03-28T15:03:54.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.038+0000 7effdd2c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7effd8072100 0x7effd806f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:54.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.038+0000 7effdd2c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effd815a8d0 con 0x7effd8154b50 2024-03-28T15:03:54.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.038+0000 7effdd2c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effd80cc210 con 0x7effd806fa60 2024-03-28T15:03:54.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.038+0000 7effdd2c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7effd8072520 con 0x7effd8072100 2024-03-28T15:03:54.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.038+0000 7effd67fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7effd8072100 0x7effd806f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:54.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.038+0000 7effd6ffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7effd8154b50 0x7effd80c9420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:43692/0 (socket says 172.21.15.186:43692) 2024-03-28T15:03:54.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.038+0000 7effd6ffd640 1 -- 172.21.15.186:0/255154535 learned_addr learned my addr 172.21.15.186:0/255154535 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:03:54.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.038+0000 7effd67fc640 1 -- 172.21.15.186:0/255154535 >> v1:172.21.15.191:6789/0 conn(0x7effd806fa60 legacy=0x7effd815add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:54.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.038+0000 7effd67fc640 1 -- 172.21.15.186:0/255154535 >> v1:172.21.15.186:6789/0 conn(0x7effd8154b50 legacy=0x7effd80c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:54.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.038+0000 7effd67fc640 1 -- 172.21.15.186:0/255154535 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd80cc4f0 con 0x7effd8072100 2024-03-28T15:03:54.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.039+0000 7effd67fc640 1 --2- 172.21.15.186:0/255154535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd8072100 0x7effd806f520 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7effcc0097a0 tx=0x7effcc02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81672a3db271fe8e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:54.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.039+0000 7effd5ffb640 1 -- 172.21.15.186:0/255154535 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effcc02f810 con 0x7effd8072100 2024-03-28T15:03:54.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.039+0000 7effd5ffb640 1 -- 172.21.15.186:0/255154535 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effcc02f9b0 con 0x7effd8072100 2024-03-28T15:03:54.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.039+0000 7effd5ffb640 1 -- 172.21.15.186:0/255154535 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effcc02fd80 con 0x7effd8072100 2024-03-28T15:03:54.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.040+0000 7effdd2c9640 1 -- 172.21.15.186:0/255154535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd8072100 msgr2=0x7effd806f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:54.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.040+0000 7effdd2c9640 1 --2- 172.21.15.186:0/255154535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd8072100 0x7effd806f520 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7effcc0097a0 tx=0x7effcc02b9f0 comp rx=0 tx=0).stop 2024-03-28T15:03:54.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.040+0000 7effdd2c9640 1 -- 172.21.15.186:0/255154535 shutdown_connections 2024-03-28T15:03:54.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.040+0000 7effdd2c9640 1 --2- 172.21.15.186:0/255154535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd8072100 0x7effd806f520 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:54.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.040+0000 7effdd2c9640 1 -- 172.21.15.186:0/255154535 >> 172.21.15.186:0/255154535 conn(0x7effd806d680 msgr2=0x7effd806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:54.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.040+0000 7effdd2c9640 1 -- 172.21.15.186:0/255154535 shutdown_connections 2024-03-28T15:03:54.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.040+0000 7effdd2c9640 1 -- 172.21.15.186:0/255154535 wait complete. 2024-03-28T15:03:54.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.041+0000 7effdd2c9640 1 Processor -- start 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.041+0000 7effdd2c9640 1 -- start start 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effdd2c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effd806fa60 0x7effd814c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effdd2c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd8072100 0x7effd815ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effdd2c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effd8154b50 0x7effd80c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effdd2c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7effd80ccd90 con 0x7effd8072100 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effdd2c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7effd80ccab0 con 0x7effd806fa60 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effdd2c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7effd8154f30 con 0x7effd8154b50 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effd67fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd8072100 0x7effd815ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effd67fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd8072100 0x7effd815ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34388/0 (socket says 172.21.15.186:34388) 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effd67fc640 1 -- 172.21.15.186:0/3925065348 learned_addr learned my addr 172.21.15.186:0/3925065348 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effd6ffd640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effd806fa60 0x7effd814c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:54.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effd77fe640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effd8154b50 0x7effd80c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:54.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effd67fc640 1 -- 172.21.15.186:0/3925065348 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effd8154b50 msgr2=0x7effd80c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:54.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effd67fc640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effd8154b50 0x7effd80c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:54.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effd67fc640 1 -- 172.21.15.186:0/3925065348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effd806fa60 msgr2=0x7effd814c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:54.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effd67fc640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effd806fa60 0x7effd814c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:54.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.042+0000 7effd67fc640 1 -- 172.21.15.186:0/3925065348 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd81f4c50 con 0x7effd8072100 2024-03-28T15:03:54.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.043+0000 7effd67fc640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd8072100 0x7effd815ba30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7effcc009890 tx=0x7effcc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:54.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.043+0000 7effbf7fe640 1 -- 172.21.15.186:0/3925065348 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effcc046070 con 0x7effd8072100 2024-03-28T15:03:54.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.043+0000 7effbf7fe640 1 -- 172.21.15.186:0/3925065348 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effcc037c90 con 0x7effd8072100 2024-03-28T15:03:54.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.043+0000 7effdd2c9640 1 -- 172.21.15.186:0/3925065348 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd81f4970 con 0x7effd8072100 2024-03-28T15:03:54.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.043+0000 7effbf7fe640 1 -- 172.21.15.186:0/3925065348 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effcc02fe60 con 0x7effd8072100 2024-03-28T15:03:54.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.043+0000 7effdd2c9640 1 -- 172.21.15.186:0/3925065348 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd814bbf0 con 0x7effd8072100 2024-03-28T15:03:54.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.044+0000 7effdd2c9640 1 -- 172.21.15.186:0/3925065348 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effd814be30 con 0x7effd8072100 2024-03-28T15:03:54.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.044+0000 7effbf7fe640 1 -- 172.21.15.186:0/3925065348 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7effcc049070 con 0x7effd8072100 2024-03-28T15:03:54.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.045+0000 7effbf7fe640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7effc406c560 0x7effc406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:54.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.045+0000 7effbf7fe640 1 -- 172.21.15.186:0/3925065348 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7effcc0ad2f0 con 0x7effd8072100 2024-03-28T15:03:54.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.045+0000 7effd6ffd640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7effc406c560 0x7effc406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:54.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.045+0000 7effd6ffd640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7effc406c560 0x7effc406ea20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7effc8005c00 tx=0x7effc8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:54.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.050+0000 7effbf7fe640 1 -- 172.21.15.186:0/3925065348 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7effd814be30 con 0x7effd8072100 2024-03-28T15:03:54.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.264+0000 7effdd2c9640 1 -- 172.21.15.186:0/3925065348 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effd8069500 con 0x7effd8072100 2024-03-28T15:03:54.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.265+0000 7effbf7fe640 1 -- 172.21.15.186:0/3925065348 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6016 (secure 0 0 0) 0x7effcc033030 con 0x7effd8072100 2024-03-28T15:03:54.265 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:54.265 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:03:51.664415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:03:54.266 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 15 2024-03-28T15:03:54.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 -- 172.21.15.186:0/3925065348 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7effc406c560 msgr2=0x7effc406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:54.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7effc406c560 0x7effc406ea20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7effc8005c00 tx=0x7effc8009040 comp rx=0 tx=0).stop 2024-03-28T15:03:54.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 -- 172.21.15.186:0/3925065348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd8072100 msgr2=0x7effd815ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:54.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd8072100 0x7effd815ba30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7effcc009890 tx=0x7effcc004730 comp rx=0 tx=0).stop 2024-03-28T15:03:54.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 -- 172.21.15.186:0/3925065348 shutdown_connections 2024-03-28T15:03:54.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7effc406c560 0x7effc406ea20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:54.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effd8154b50 0x7effd80c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:54.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd8072100 0x7effd815ba30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:54.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 --2- 172.21.15.186:0/3925065348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effd806fa60 0x7effd814c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:54.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 -- 172.21.15.186:0/3925065348 >> 172.21.15.186:0/3925065348 conn(0x7effd806d680 msgr2=0x7effd815acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:54.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 -- 172.21.15.186:0/3925065348 shutdown_connections 2024-03-28T15:03:54.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:54.267+0000 7effdd2c9640 1 -- 172.21.15.186:0/3925065348 wait complete. 2024-03-28T15:03:54.278 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:03:55.585 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:03:55.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:03:56.279 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:56.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.391+0000 7f88bbc47640 1 Processor -- start 2024-03-28T15:03:56.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.391+0000 7f88bbc47640 1 -- start start 2024-03-28T15:03:56.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.392+0000 7f88bbc47640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f88b4154bd0 0x7f88b415a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:56.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.392+0000 7f88bbc47640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88b40c6e00 con 0x7f88b40c53c0 2024-03-28T15:03:56.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.392+0000 7f88bbc47640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88b415b080 con 0x7f88b40c57e0 2024-03-28T15:03:56.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.392+0000 7f88bbc47640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f88b40c5ca0 con 0x7f88b4154bd0 2024-03-28T15:03:56.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.392+0000 7f88b91e2640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f88b40c53c0 0x7f88b40c66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:43700/0 (socket says 172.21.15.186:43700) 2024-03-28T15:03:56.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.392+0000 7f88b91e2640 1 -- 172.21.15.186:0/607231845 learned_addr learned my addr 172.21.15.186:0/607231845 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:03:56.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.392+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3847580215 0 0) 0x7f88b40c6e00 con 0x7f88b40c53c0 2024-03-28T15:03:56.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.392+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88a0003520 con 0x7f88b40c53c0 2024-03-28T15:03:56.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.392+0000 7f88b99e3640 1 --2- 172.21.15.186:0/607231845 >> v2:172.21.15.191:3300/0 conn(0x7f88b4154bd0 0x7f88b415a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:56.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3373692709 0 0) 0x7f88a0003520 con 0x7f88b40c53c0 2024-03-28T15:03:56.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 >> v2:172.21.15.191:3300/0 conn(0x7f88b4154bd0 msgr2=0x7f88b415a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:56.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 --2- 172.21.15.186:0/607231845 >> v2:172.21.15.191:3300/0 conn(0x7f88b4154bd0 0x7f88b415a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:56.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 >> v1:172.21.15.191:6789/0 conn(0x7f88b40c57e0 legacy=0x7f88b40cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:56.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b415b360 con 0x7f88b40c53c0 2024-03-28T15:03:56.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f88a4001670 con 0x7f88b40c53c0 2024-03-28T15:03:56.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 >> v1:172.21.15.186:6789/0 conn(0x7f88b40c53c0 legacy=0x7f88b40c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:56.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88a00039b0 0x7f88a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88a0004800 0x7f88a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88a00051c0 0x7f88a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f88b40c5ca0 con 0x7f88a0004800 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f88a000c070 con 0x7f88a00051c0 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f88a000c270 con 0x7f88a00039b0 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b91e2640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88a0004800 0x7f88a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b99e3640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88a00039b0 0x7f88a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88ba1e4640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88a00051c0 0x7f88a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b91e2640 1 -- 172.21.15.186:0/607231845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88a00039b0 msgr2=0x7f88a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b91e2640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88a00039b0 0x7f88a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b91e2640 1 -- 172.21.15.186:0/607231845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88a00051c0 msgr2=0x7f88a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b91e2640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88a00051c0 0x7f88a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88b91e2640 1 -- 172.21.15.186:0/607231845 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a0003520 con 0x7f88a0004800 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88ba1e4640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88a00051c0 0x7f88a000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:03:56.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.393+0000 7f88ba1e4640 1 -- 172.21.15.186:0/607231845 reap_dead start 2024-03-28T15:03:56.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.394+0000 7f88b91e2640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88a0004800 0x7f88a0004c80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f88a400c240 tx=0x7f88a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81966aab6e12a6e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:56.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.394+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4007d10 con 0x7f88a0004800 2024-03-28T15:03:56.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.394+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a4030070 con 0x7f88a0004800 2024-03-28T15:03:56.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.394+0000 7f88b89e1640 1 -- 172.21.15.186:0/607231845 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a402cdc0 con 0x7f88a0004800 2024-03-28T15:03:56.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.395+0000 7f88bbc47640 1 -- 172.21.15.186:0/607231845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88a0004800 msgr2=0x7f88a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:56.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.395+0000 7f88bbc47640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88a0004800 0x7f88a0004c80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f88a400c240 tx=0x7f88a0005830 comp rx=0 tx=0).stop 2024-03-28T15:03:56.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.395+0000 7f88bbc47640 1 -- 172.21.15.186:0/607231845 shutdown_connections 2024-03-28T15:03:56.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.395+0000 7f88bbc47640 1 --2- 172.21.15.186:0/607231845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88a0004800 0x7f88a0004c80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:56.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.395+0000 7f88bbc47640 1 -- 172.21.15.186:0/607231845 >> 172.21.15.186:0/607231845 conn(0x7f88b40731f0 msgr2=0x7f88b4073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:56.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.395+0000 7f88bbc47640 1 -- 172.21.15.186:0/607231845 shutdown_connections 2024-03-28T15:03:56.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.395+0000 7f88bbc47640 1 -- 172.21.15.186:0/607231845 wait complete. 2024-03-28T15:03:56.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.396+0000 7f88bbc47640 1 Processor -- start 2024-03-28T15:03:56.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.396+0000 7f88bbc47640 1 -- start start 2024-03-28T15:03:56.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88bbc47640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88a0004800 0x7f88b40cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:56.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88bbc47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88b414abe0 0x7f88b4144a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:56.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88bbc47640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88b4144f50 0x7f88b41f0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:56.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88bbc47640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f88b40c61d0 con 0x7f88b414abe0 2024-03-28T15:03:56.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88bbc47640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f88b40c5ef0 con 0x7f88b4144f50 2024-03-28T15:03:56.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88bbc47640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f88b40cc390 con 0x7f88a0004800 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88b91e2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88b414abe0 0x7f88b4144a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88b91e2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88b414abe0 0x7f88b4144a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34418/0 (socket says 172.21.15.186:34418) 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88b91e2640 1 -- 172.21.15.186:0/2957297233 learned_addr learned my addr 172.21.15.186:0/2957297233 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88ba1e4640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88b4144f50 0x7f88b41f0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88b99e3640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88a0004800 0x7f88b40cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88b91e2640 1 -- 172.21.15.186:0/2957297233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88a0004800 msgr2=0x7f88b40cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88b91e2640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88a0004800 0x7f88b40cd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88b91e2640 1 -- 172.21.15.186:0/2957297233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88b4144f50 msgr2=0x7f88b41f0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88b91e2640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88b4144f50 0x7f88b41f0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88b91e2640 1 -- 172.21.15.186:0/2957297233 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b41f3680 con 0x7f88b414abe0 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.397+0000 7f88b91e2640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88b414abe0 0x7f88b4144a10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f88a4001970 tx=0x7f88b40d0c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.398+0000 7f88aa7fc640 1 -- 172.21.15.186:0/2957297233 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4045070 con 0x7f88b414abe0 2024-03-28T15:03:56.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.398+0000 7f88aa7fc640 1 -- 172.21.15.186:0/2957297233 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a40049f0 con 0x7f88b414abe0 2024-03-28T15:03:56.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.398+0000 7f88bbc47640 1 -- 172.21.15.186:0/2957297233 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b41f3960 con 0x7f88b414abe0 2024-03-28T15:03:56.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.398+0000 7f88aa7fc640 1 -- 172.21.15.186:0/2957297233 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4007d10 con 0x7f88b414abe0 2024-03-28T15:03:56.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.398+0000 7f88bbc47640 1 -- 172.21.15.186:0/2957297233 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b41f3f20 con 0x7f88b414abe0 2024-03-28T15:03:56.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.399+0000 7f88bbc47640 1 -- 172.21.15.186:0/2957297233 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88b4145360 con 0x7f88b414abe0 2024-03-28T15:03:56.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.399+0000 7f88aa7fc640 1 -- 172.21.15.186:0/2957297233 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f88a4007070 con 0x7f88b414abe0 2024-03-28T15:03:56.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.400+0000 7f88aa7fc640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f889806c530 0x7f889806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:56.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.400+0000 7f88b99e3640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f889806c530 0x7f889806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:56.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.400+0000 7f88aa7fc640 1 -- 172.21.15.186:0/2957297233 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7f88a40ac440 con 0x7f88b414abe0 2024-03-28T15:03:56.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.400+0000 7f88b99e3640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f889806c530 0x7f889806e9f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f88b0007520 tx=0x7f88b000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:56.404 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.404+0000 7f88aa7fc640 1 -- 172.21.15.186:0/2957297233 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f88a407a8b0 con 0x7f88b414abe0 2024-03-28T15:03:56.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.619+0000 7f88bbc47640 1 -- 172.21.15.186:0/2957297233 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88b4069500 con 0x7f88b414abe0 2024-03-28T15:03:56.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.619+0000 7f88aa7fc640 1 -- 172.21.15.186:0/2957297233 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6016 (secure 0 0 0) 0x7f88a407a1b0 con 0x7f88b414abe0 2024-03-28T15:03:56.620 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:56.620 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:03:51.664415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:03:56.620 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 15 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 -- 172.21.15.186:0/2957297233 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f889806c530 msgr2=0x7f889806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f889806c530 0x7f889806e9f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f88b0007520 tx=0x7f88b000b040 comp rx=0 tx=0).stop 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 -- 172.21.15.186:0/2957297233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88b414abe0 msgr2=0x7f88b4144a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88b414abe0 0x7f88b4144a10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f88a4001970 tx=0x7f88b40d0c50 comp rx=0 tx=0).stop 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 -- 172.21.15.186:0/2957297233 shutdown_connections 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f889806c530 0x7f889806e9f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88b4144f50 0x7f88b41f0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88b414abe0 0x7f88b4144a10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 --2- 172.21.15.186:0/2957297233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88a0004800 0x7f88b40cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 -- 172.21.15.186:0/2957297233 >> 172.21.15.186:0/2957297233 conn(0x7f88b40731f0 msgr2=0x7f88b4159910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 -- 172.21.15.186:0/2957297233 shutdown_connections 2024-03-28T15:03:56.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:56.621+0000 7f88bbc47640 1 -- 172.21.15.186:0/2957297233 wait complete. 2024-03-28T15:03:56.632 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:03:58.634 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:03:58.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.747+0000 7f3b0aef4640 1 Processor -- start 2024-03-28T15:03:58.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.747+0000 7f3b0aef4640 1 -- start start 2024-03-28T15:03:58.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.747+0000 7f3b0aef4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3b04154bd0 0x7f3b0415a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:58.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.747+0000 7f3b0aef4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3b040c53c0 0x7f3b040c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:58.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.747+0000 7f3b0aef4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b0415b080 con 0x7f3b040c57e0 2024-03-28T15:03:58.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.747+0000 7f3b0aef4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f3b040c5ca0 con 0x7f3b04154bd0 2024-03-28T15:03:58.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.747+0000 7f3b0aef4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f3b0415b360 con 0x7f3b040c53c0 2024-03-28T15:03:58.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.748+0000 7f3b08c90640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3b04154bd0 0x7f3b0415a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:58.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.748+0000 7f3b08c90640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3b04154bd0 0x7f3b0415a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44988/0 (socket says 172.21.15.186:44988) 2024-03-28T15:03:58.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.748+0000 7f3b08c90640 1 -- 172.21.15.186:0/592546289 learned_addr learned my addr 172.21.15.186:0/592546289 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:58.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.748+0000 7f3b03fff640 1 --2- 172.21.15.186:0/592546289 >> v2:172.21.15.191:3300/0 conn(0x7f3b040c53c0 0x7f3b040c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:58.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.748+0000 7f3b08c90640 1 -- 172.21.15.186:0/592546289 >> v2:172.21.15.191:3300/0 conn(0x7f3b040c53c0 msgr2=0x7f3b040c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:58.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.748+0000 7f3b08c90640 1 --2- 172.21.15.186:0/592546289 >> v2:172.21.15.191:3300/0 conn(0x7f3b040c53c0 0x7f3b040c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:58.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.748+0000 7f3b08c90640 1 -- 172.21.15.186:0/592546289 >> v1:172.21.15.191:6789/0 conn(0x7f3b040c57e0 legacy=0x7f3b040cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:03:58.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.748+0000 7f3b08c90640 1 -- 172.21.15.186:0/592546289 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0415b640 con 0x7f3b04154bd0 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.749+0000 7f3b08c90640 1 --2- 172.21.15.186:0/592546289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b04154bd0 0x7f3b0415a7d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3b0408d6b0 tx=0x7f3afc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bcbbe744b5c0fef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.749+0000 7f3b037fe640 1 -- 172.21.15.186:0/592546289 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc02cb20 con 0x7f3b04154bd0 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.749+0000 7f3b037fe640 1 -- 172.21.15.186:0/592546289 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3afc02ccc0 con 0x7f3b04154bd0 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.749+0000 7f3b037fe640 1 -- 172.21.15.186:0/592546289 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc034610 con 0x7f3b04154bd0 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.749+0000 7f3b0aef4640 1 -- 172.21.15.186:0/592546289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b04154bd0 msgr2=0x7f3b0415a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.749+0000 7f3b0aef4640 1 --2- 172.21.15.186:0/592546289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b04154bd0 0x7f3b0415a7d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3b0408d6b0 tx=0x7f3afc02bf80 comp rx=0 tx=0).stop 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.749+0000 7f3b0aef4640 1 -- 172.21.15.186:0/592546289 shutdown_connections 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.749+0000 7f3b0aef4640 1 --2- 172.21.15.186:0/592546289 >> v2:172.21.15.191:3300/0 conn(0x7f3b040c53c0 0x7f3b040c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.749+0000 7f3b0aef4640 1 --2- 172.21.15.186:0/592546289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b04154bd0 0x7f3b0415a7d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.750+0000 7f3b0aef4640 1 -- 172.21.15.186:0/592546289 >> 172.21.15.186:0/592546289 conn(0x7f3b040731f0 msgr2=0x7f3b04073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.750+0000 7f3b0aef4640 1 -- 172.21.15.186:0/592546289 shutdown_connections 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.750+0000 7f3b0aef4640 1 -- 172.21.15.186:0/592546289 wait complete. 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.751+0000 7f3b0aef4640 1 Processor -- start 2024-03-28T15:03:58.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.751+0000 7f3b0aef4640 1 -- start start 2024-03-28T15:03:58.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b0aef4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b040c53c0 0x7f3b0415c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:58.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b0aef4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b040c57e0 0x7f3b0415d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:58.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b0aef4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04154bd0 0x7f3b04153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:58.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b0aef4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b040cc390 con 0x7f3b040c53c0 2024-03-28T15:03:58.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b0aef4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3b040c61d0 con 0x7f3b040c57e0 2024-03-28T15:03:58.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b0aef4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3b0415ad10 con 0x7f3b04154bd0 2024-03-28T15:03:58.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b08c90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b040c53c0 0x7f3b0415c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:58.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b08c90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b040c53c0 0x7f3b0415c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44990/0 (socket says 172.21.15.186:44990) 2024-03-28T15:03:58.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b08c90640 1 -- 172.21.15.186:0/2322899446 learned_addr learned my addr 172.21.15.186:0/2322899446 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:03:58.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b03fff640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b040c57e0 0x7f3b0415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:58.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b09491640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04154bd0 0x7f3b04153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b08c90640 1 -- 172.21.15.186:0/2322899446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04154bd0 msgr2=0x7f3b04153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b08c90640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04154bd0 0x7f3b04153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b08c90640 1 -- 172.21.15.186:0/2322899446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b040c57e0 msgr2=0x7f3b0415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b08c90640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b040c57e0 0x7f3b0415d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b08c90640 1 -- 172.21.15.186:0/2322899446 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0415cb50 con 0x7f3b040c53c0 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b03fff640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b040c57e0 0x7f3b0415d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b08c90640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b040c53c0 0x7f3b0415c610 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3afc000f80 tx=0x7f3afc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b017fa640 1 -- 172.21.15.186:0/2322899446 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc043020 con 0x7f3b040c53c0 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b017fa640 1 -- 172.21.15.186:0/2322899446 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3afc03e450 con 0x7f3b040c53c0 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b0aef4640 1 -- 172.21.15.186:0/2322899446 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b0415dd80 con 0x7f3b040c53c0 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b017fa640 1 -- 172.21.15.186:0/2322899446 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc03e840 con 0x7f3b040c53c0 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.752+0000 7f3b0aef4640 1 -- 172.21.15.186:0/2322899446 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b0415e960 con 0x7f3b040c53c0 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.753+0000 7f3b0aef4640 1 -- 172.21.15.186:0/2322899446 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b04069500 con 0x7f3b040c53c0 2024-03-28T15:03:58.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.754+0000 7f3b017fa640 1 -- 172.21.15.186:0/2322899446 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3afc046020 con 0x7f3b040c53c0 2024-03-28T15:03:58.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.754+0000 7f3b017fa640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ae406c560 0x7f3ae406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:03:58.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.754+0000 7f3b03fff640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ae406c560 0x7f3ae406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:03:58.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.754+0000 7f3b017fa640 1 -- 172.21.15.186:0/2322899446 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7f3afc0ade00 con 0x7f3b040c53c0 2024-03-28T15:03:58.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.755+0000 7f3b03fff640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ae406c560 0x7f3ae406ea20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3b040b5160 tx=0x7f3aec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:03:58.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.759+0000 7f3b017fa640 1 -- 172.21.15.186:0/2322899446 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3afc04d070 con 0x7f3b040c53c0 2024-03-28T15:03:58.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.971+0000 7f3b0aef4640 1 -- 172.21.15.186:0/2322899446 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b04159670 con 0x7f3b040c53c0 2024-03-28T15:03:58.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.972+0000 7f3b017fa640 1 -- 172.21.15.186:0/2322899446 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6013 (secure 0 0 0) 0x7f3afc07c390 con 0x7f3b040c53c0 2024-03-28T15:03:58.973 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:03:58.973 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:03:57.442151+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:03:58.973 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 16 2024-03-28T15:03:58.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 -- 172.21.15.186:0/2322899446 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ae406c560 msgr2=0x7f3ae406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:58.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ae406c560 0x7f3ae406ea20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3b040b5160 tx=0x7f3aec008040 comp rx=0 tx=0).stop 2024-03-28T15:03:58.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 -- 172.21.15.186:0/2322899446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b040c53c0 msgr2=0x7f3b0415c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:03:58.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b040c53c0 0x7f3b0415c610 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3afc000f80 tx=0x7f3afc004a40 comp rx=0 tx=0).stop 2024-03-28T15:03:58.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 -- 172.21.15.186:0/2322899446 shutdown_connections 2024-03-28T15:03:58.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ae406c560 0x7f3ae406ea20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:58.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04154bd0 0x7f3b04153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:58.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b040c57e0 0x7f3b0415d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:58.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 --2- 172.21.15.186:0/2322899446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b040c53c0 0x7f3b0415c610 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:03:58.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 -- 172.21.15.186:0/2322899446 >> 172.21.15.186:0/2322899446 conn(0x7f3b040731f0 msgr2=0x7f3b040ce030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:03:58.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 -- 172.21.15.186:0/2322899446 shutdown_connections 2024-03-28T15:03:58.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:03:58.974+0000 7f3b0aef4640 1 -- 172.21.15.186:0/2322899446 wait complete. 2024-03-28T15:03:58.986 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 16, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T15:01:31.425299+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 16, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4879": { "addr": "172.21.15.191:6837/3828930138", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3828930138, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3828930138, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4879, "incarnation": 14, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:rejoin", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T15:03:57.442151+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4879 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/1278716016", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6833", "nonce": 1278716016, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1278716016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6835/2068761001", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2068761001, "type": "v2" }, { "addr": "172.21.15.186:6835", "nonce": 2068761001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2344737463", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6837", "nonce": 2344737463, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2344737463, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4284, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/1548903381", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 1548903381, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 1548903381, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T15:04:00.987 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:04:01.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.101+0000 7f69a070f640 1 Processor -- start 2024-03-28T15:04:01.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.101+0000 7f69a070f640 1 -- start start 2024-03-28T15:04:01.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f69a070f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6998151990 0x7f69980c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:01.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f69a070f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980c2d00 0x7f69980c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:01.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f69a070f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f699806caa0 con 0x7f69980b1e30 2024-03-28T15:04:01.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f69a070f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f69980c1da0 con 0x7f6998151990 2024-03-28T15:04:01.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f69a070f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f699806cd80 con 0x7f69980c2d00 2024-03-28T15:04:01.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f699e4ab640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6998151990 0x7f69980c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:01.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f699e4ab640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6998151990 0x7f69980c2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:45006/0 (socket says 172.21.15.186:45006) 2024-03-28T15:04:01.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f699e4ab640 1 -- 172.21.15.186:0/888843579 learned_addr learned my addr 172.21.15.186:0/888843579 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:04:01.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f699ecac640 1 --2- 172.21.15.186:0/888843579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980c2d00 0x7f69980c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:01.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f699e4ab640 1 -- 172.21.15.186:0/888843579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980c2d00 msgr2=0x7f69980c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:04:01.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f699e4ab640 1 --2- 172.21.15.186:0/888843579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980c2d00 0x7f69980c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:01.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f699e4ab640 1 -- 172.21.15.186:0/888843579 >> v1:172.21.15.191:6789/0 conn(0x7f69980b1e30 legacy=0x7f69980712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:01.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f699e4ab640 1 -- 172.21.15.186:0/888843579 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69980c1340 con 0x7f6998151990 2024-03-28T15:04:01.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.102+0000 7f699ecac640 1 --2- 172.21.15.186:0/888843579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980c2d00 0x7f69980c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:04:01.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.103+0000 7f699e4ab640 1 --2- 172.21.15.186:0/888843579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6998151990 0x7f69980c2230 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f699000bb10 tx=0x7f699002e1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f357f248586aee34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:01.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.103+0000 7f699d4a9640 1 -- 172.21.15.186:0/888843579 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699003b070 con 0x7f6998151990 2024-03-28T15:04:01.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.103+0000 7f699d4a9640 1 -- 172.21.15.186:0/888843579 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6990036440 con 0x7f6998151990 2024-03-28T15:04:01.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.103+0000 7f699d4a9640 1 -- 172.21.15.186:0/888843579 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990036810 con 0x7f6998151990 2024-03-28T15:04:01.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.104+0000 7f69a070f640 1 -- 172.21.15.186:0/888843579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6998151990 msgr2=0x7f69980c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.104+0000 7f69a070f640 1 --2- 172.21.15.186:0/888843579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6998151990 0x7f69980c2230 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f699000bb10 tx=0x7f699002e1f0 comp rx=0 tx=0).stop 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.104+0000 7f69a070f640 1 -- 172.21.15.186:0/888843579 shutdown_connections 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.104+0000 7f69a070f640 1 --2- 172.21.15.186:0/888843579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980c2d00 0x7f69980c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.104+0000 7f69a070f640 1 --2- 172.21.15.186:0/888843579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6998151990 0x7f69980c2230 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.104+0000 7f69a070f640 1 -- 172.21.15.186:0/888843579 >> 172.21.15.186:0/888843579 conn(0x7f69980c8a20 msgr2=0x7f69980c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.104+0000 7f69a070f640 1 -- 172.21.15.186:0/888843579 shutdown_connections 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.104+0000 7f69a070f640 1 -- 172.21.15.186:0/888843579 wait complete. 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.105+0000 7f69a070f640 1 Processor -- start 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.105+0000 7f69a070f640 1 -- start start 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.105+0000 7f69a070f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980b1e30 0x7f69981ed110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.105+0000 7f69a070f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980c2d00 0x7f69981571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.105+0000 7f69a070f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6998151990 0x7f6998155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f69a070f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6998073dd0 con 0x7f6998151990 2024-03-28T15:04:01.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f69a070f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f69980c1900 con 0x7f69980c2d00 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f69a070f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f6998073f50 con 0x7f69980b1e30 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f699ecac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6998151990 0x7f6998155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f699ecac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6998151990 0x7f6998155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:45010/0 (socket says 172.21.15.186:45010) 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f699ecac640 1 -- 172.21.15.186:0/146914270 learned_addr learned my addr 172.21.15.186:0/146914270 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f699e4ab640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980b1e30 0x7f69981ed110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f699dcaa640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980c2d00 0x7f69981571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f699ecac640 1 -- 172.21.15.186:0/146914270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980b1e30 msgr2=0x7f69981ed110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f699ecac640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980b1e30 0x7f69981ed110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f699ecac640 1 -- 172.21.15.186:0/146914270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980c2d00 msgr2=0x7f69981571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f699ecac640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980c2d00 0x7f69981571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f699ecac640 1 -- 172.21.15.186:0/146914270 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6998157700 con 0x7f6998151990 2024-03-28T15:04:01.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f699ecac640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6998151990 0x7f6998155550 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6994007e70 tx=0x7f699400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f698effd640 1 -- 172.21.15.186:0/146914270 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699401a070 con 0x7f6998151990 2024-03-28T15:04:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f698effd640 1 -- 172.21.15.186:0/146914270 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6994003bb0 con 0x7f6998151990 2024-03-28T15:04:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f69a070f640 1 -- 172.21.15.186:0/146914270 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69981579c0 con 0x7f6998151990 2024-03-28T15:04:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f698effd640 1 -- 172.21.15.186:0/146914270 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6994004030 con 0x7f6998151990 2024-03-28T15:04:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.106+0000 7f69a070f640 1 -- 172.21.15.186:0/146914270 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69980c3ac0 con 0x7f6998151990 2024-03-28T15:04:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.108+0000 7f69a070f640 1 -- 172.21.15.186:0/146914270 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6998069500 con 0x7f6998151990 2024-03-28T15:04:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.108+0000 7f698effd640 1 -- 172.21.15.186:0/146914270 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f699401d070 con 0x7f6998151990 2024-03-28T15:04:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.108+0000 7f698effd640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f697c06c530 0x7f697c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.108+0000 7f698effd640 1 -- 172.21.15.186:0/146914270 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7f699408b9c0 con 0x7f6998151990 2024-03-28T15:04:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.108+0000 7f699e4ab640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f697c06c530 0x7f697c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:01.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.109+0000 7f699e4ab640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f697c06c530 0x7f697c06e9f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6990038000 tx=0x7f6990032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:01.114 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.113+0000 7f698effd640 1 -- 172.21.15.186:0/146914270 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f699408de20 con 0x7f6998151990 2024-03-28T15:04:01.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.324+0000 7f69a070f640 1 -- 172.21.15.186:0/146914270 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69980b1a00 con 0x7f6998151990 2024-03-28T15:04:01.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.325+0000 7f698effd640 1 -- 172.21.15.186:0/146914270 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6013 (secure 0 0 0) 0x7f6994010070 con 0x7f6998151990 2024-03-28T15:04:01.325 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:04:01.326 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:03:57.442151+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:04:01.326 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 16 2024-03-28T15:04:01.327 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 -- 172.21.15.186:0/146914270 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f697c06c530 msgr2=0x7f697c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:01.327 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f697c06c530 0x7f697c06e9f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6990038000 tx=0x7f6990032000 comp rx=0 tx=0).stop 2024-03-28T15:04:01.327 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 -- 172.21.15.186:0/146914270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6998151990 msgr2=0x7f6998155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:01.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6998151990 0x7f6998155550 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6994007e70 tx=0x7f699400d4a0 comp rx=0 tx=0).stop 2024-03-28T15:04:01.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 -- 172.21.15.186:0/146914270 shutdown_connections 2024-03-28T15:04:01.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f697c06c530 0x7f697c06e9f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:01.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6998151990 0x7f6998155550 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:01.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980c2d00 0x7f69981571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:01.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 --2- 172.21.15.186:0/146914270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69980b1e30 0x7f69981ed110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:01.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 -- 172.21.15.186:0/146914270 >> 172.21.15.186:0/146914270 conn(0x7f69980c8a20 msgr2=0x7f69980713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:01.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 -- 172.21.15.186:0/146914270 shutdown_connections 2024-03-28T15:04:01.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:01.327+0000 7f69a070f640 1 -- 172.21.15.186:0/146914270 wait complete. 2024-03-28T15:04:01.338 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:04:03.339 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:04:03.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.451+0000 7fa25e0a4640 1 Processor -- start 2024-03-28T15:04:03.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.451+0000 7fa25e0a4640 1 -- start start 2024-03-28T15:04:03.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa25e0a4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa258152810 0x7fa258158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:03.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa25e0a4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa258071340 0x7fa25806c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:03.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa25e0a4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa25806ce60 0x7fa2580c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:03.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa25e0a4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa258159500 con 0x7fa258071340 2024-03-28T15:04:03.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa25e0a4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fa258072df0 con 0x7fa25806ce60 2024-03-28T15:04:03.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa25e0a4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa2580c8660 con 0x7fa258152810 2024-03-28T15:04:03.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa256ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa258071340 0x7fa25806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:03.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa256ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa258071340 0x7fa25806c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:45026/0 (socket says 172.21.15.186:45026) 2024-03-28T15:04:03.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa256ffd640 1 -- 172.21.15.186:0/3651218217 learned_addr learned my addr 172.21.15.186:0/3651218217 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:04:03.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa2577fe640 1 --2- 172.21.15.186:0/3651218217 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa258152810 0x7fa258158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:03.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa257fff640 1 --2- 172.21.15.186:0/3651218217 >> v2:172.21.15.191:3300/0 conn(0x7fa25806ce60 0x7fa2580c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:03.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa256ffd640 1 -- 172.21.15.186:0/3651218217 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa258152810 msgr2=0x7fa258158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:04:03.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa256ffd640 1 --2- 172.21.15.186:0/3651218217 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa258152810 0x7fa258158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa256ffd640 1 -- 172.21.15.186:0/3651218217 >> v2:172.21.15.191:3300/0 conn(0x7fa25806ce60 msgr2=0x7fa2580c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:04:03.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.452+0000 7fa256ffd640 1 --2- 172.21.15.186:0/3651218217 >> v2:172.21.15.191:3300/0 conn(0x7fa25806ce60 0x7fa2580c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.453+0000 7fa256ffd640 1 -- 172.21.15.186:0/3651218217 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa258071d50 con 0x7fa258071340 2024-03-28T15:04:03.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.453+0000 7fa257fff640 1 --2- 172.21.15.186:0/3651218217 >> v2:172.21.15.191:3300/0 conn(0x7fa25806ce60 0x7fa2580c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:04:03.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.453+0000 7fa2577fe640 1 --2- 172.21.15.186:0/3651218217 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa258152810 0x7fa258158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:04:03.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.453+0000 7fa256ffd640 1 --2- 172.21.15.186:0/3651218217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa258071340 0x7fa25806c920 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa2580b2290 tx=0x7fa24c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76910dc97ffdda9c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:03.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.453+0000 7fa2567fc640 1 -- 172.21.15.186:0/3651218217 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c035020 con 0x7fa258071340 2024-03-28T15:04:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.453+0000 7fa2567fc640 1 -- 172.21.15.186:0/3651218217 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa24c031870 con 0x7fa258071340 2024-03-28T15:04:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.453+0000 7fa2567fc640 1 -- 172.21.15.186:0/3651218217 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c031c40 con 0x7fa258071340 2024-03-28T15:04:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.454+0000 7fa25e0a4640 1 -- 172.21.15.186:0/3651218217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa258071340 msgr2=0x7fa25806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.454+0000 7fa25e0a4640 1 --2- 172.21.15.186:0/3651218217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa258071340 0x7fa25806c920 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa2580b2290 tx=0x7fa24c02b910 comp rx=0 tx=0).stop 2024-03-28T15:04:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.454+0000 7fa25e0a4640 1 -- 172.21.15.186:0/3651218217 shutdown_connections 2024-03-28T15:04:03.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.454+0000 7fa25e0a4640 1 --2- 172.21.15.186:0/3651218217 >> v2:172.21.15.191:3300/0 conn(0x7fa25806ce60 0x7fa2580c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.454+0000 7fa25e0a4640 1 --2- 172.21.15.186:0/3651218217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa258071340 0x7fa25806c920 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.454+0000 7fa25e0a4640 1 --2- 172.21.15.186:0/3651218217 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa258152810 0x7fa258158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.454+0000 7fa25e0a4640 1 -- 172.21.15.186:0/3651218217 >> 172.21.15.186:0/3651218217 conn(0x7fa2580c8a20 msgr2=0x7fa2580c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:03.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.454+0000 7fa25e0a4640 1 -- 172.21.15.186:0/3651218217 shutdown_connections 2024-03-28T15:04:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.454+0000 7fa25e0a4640 1 -- 172.21.15.186:0/3651218217 wait complete. 2024-03-28T15:04:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.455+0000 7fa25e0a4640 1 Processor -- start 2024-03-28T15:04:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa25e0a4640 1 -- start start 2024-03-28T15:04:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa25e0a4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25806ce60 0x7fa258156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:03.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa25e0a4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa258071340 0x7fa25814b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:03.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa25e0a4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa258152810 0x7fa25814bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:03.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa25e0a4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2580725f0 con 0x7fa258152810 2024-03-28T15:04:03.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa25e0a4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa258072310 con 0x7fa258071340 2024-03-28T15:04:03.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa25e0a4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa2580728d0 con 0x7fa25806ce60 2024-03-28T15:04:03.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa257fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa258152810 0x7fa25814bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:03.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa257fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa258152810 0x7fa25814bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:45040/0 (socket says 172.21.15.186:45040) 2024-03-28T15:04:03.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa257fff640 1 -- 172.21.15.186:0/2037964576 learned_addr learned my addr 172.21.15.186:0/2037964576 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:04:03.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa2577fe640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25806ce60 0x7fa258156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:03.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.456+0000 7fa256ffd640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa258071340 0x7fa25814b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:03.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.457+0000 7fa257fff640 1 -- 172.21.15.186:0/2037964576 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25806ce60 msgr2=0x7fa258156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.457+0000 7fa257fff640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25806ce60 0x7fa258156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.457+0000 7fa257fff640 1 -- 172.21.15.186:0/2037964576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa258071340 msgr2=0x7fa25814b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.457+0000 7fa257fff640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa258071340 0x7fa25814b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.457+0000 7fa257fff640 1 -- 172.21.15.186:0/2037964576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25814df30 con 0x7fa258152810 2024-03-28T15:04:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.457+0000 7fa257fff640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa258152810 0x7fa25814bce0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa250007e40 tx=0x7fa25000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:03.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.458+0000 7fa23ffff640 1 -- 172.21.15.186:0/2037964576 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25001a070 con 0x7fa258152810 2024-03-28T15:04:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.458+0000 7fa23ffff640 1 -- 172.21.15.186:0/2037964576 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa250003bb0 con 0x7fa258152810 2024-03-28T15:04:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.458+0000 7fa25e0a4640 1 -- 172.21.15.186:0/2037964576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25814e1f0 con 0x7fa258152810 2024-03-28T15:04:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.458+0000 7fa23ffff640 1 -- 172.21.15.186:0/2037964576 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25000de20 con 0x7fa258152810 2024-03-28T15:04:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.458+0000 7fa25e0a4640 1 -- 172.21.15.186:0/2037964576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2581541b0 con 0x7fa258152810 2024-03-28T15:04:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.459+0000 7fa25e0a4640 1 -- 172.21.15.186:0/2037964576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa258069500 con 0x7fa258152810 2024-03-28T15:04:03.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.459+0000 7fa23ffff640 1 -- 172.21.15.186:0/2037964576 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa25001d070 con 0x7fa258152810 2024-03-28T15:04:03.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.460+0000 7fa23ffff640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa24406c530 0x7fa24406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:03.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.460+0000 7fa23ffff640 1 -- 172.21.15.186:0/2037964576 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7fa25008bac0 con 0x7fa258152810 2024-03-28T15:04:03.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.460+0000 7fa2577fe640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa24406c530 0x7fa24406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:03.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.460+0000 7fa2577fe640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa24406c530 0x7fa24406e9f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa248006e80 tx=0x7fa248008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:03.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.464+0000 7fa23ffff640 1 -- 172.21.15.186:0/2037964576 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa250010070 con 0x7fa258152810 2024-03-28T15:04:03.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.679+0000 7fa25e0a4640 1 -- 172.21.15.186:0/2037964576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2580c77f0 con 0x7fa258152810 2024-03-28T15:04:03.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.679+0000 7fa23ffff640 1 -- 172.21.15.186:0/2037964576 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6013 (secure 0 0 0) 0x7fa250059860 con 0x7fa258152810 2024-03-28T15:04:03.680 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:04:03.681 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:03:57.442151+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:04:03.681 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 16 2024-03-28T15:04:03.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.681+0000 7fa25e0a4640 1 -- 172.21.15.186:0/2037964576 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa24406c530 msgr2=0x7fa24406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:03.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.681+0000 7fa25e0a4640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa24406c530 0x7fa24406e9f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa248006e80 tx=0x7fa248008040 comp rx=0 tx=0).stop 2024-03-28T15:04:03.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.681+0000 7fa25e0a4640 1 -- 172.21.15.186:0/2037964576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa258152810 msgr2=0x7fa25814bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:03.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.681+0000 7fa25e0a4640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa258152810 0x7fa25814bce0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa250007e40 tx=0x7fa25000d4a0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.681+0000 7fa25e0a4640 1 -- 172.21.15.186:0/2037964576 shutdown_connections 2024-03-28T15:04:03.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.681+0000 7fa25e0a4640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa24406c530 0x7fa24406e9f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.681+0000 7fa25e0a4640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa258152810 0x7fa25814bce0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.681+0000 7fa25e0a4640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa258071340 0x7fa25814b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.681+0000 7fa25e0a4640 1 --2- 172.21.15.186:0/2037964576 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25806ce60 0x7fa258156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:03.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.681+0000 7fa25e0a4640 1 -- 172.21.15.186:0/2037964576 >> 172.21.15.186:0/2037964576 conn(0x7fa2580c8a20 msgr2=0x7fa2580c39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:03.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.682+0000 7fa25e0a4640 1 -- 172.21.15.186:0/2037964576 shutdown_connections 2024-03-28T15:04:03.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:03.682+0000 7fa25e0a4640 1 -- 172.21.15.186:0/2037964576 wait complete. 2024-03-28T15:04:03.693 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:04:05.694 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:04:05.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.807+0000 7f631f047640 1 Processor -- start 2024-03-28T15:04:05.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.807+0000 7f631f047640 1 -- start start 2024-03-28T15:04:05.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.807+0000 7f631f047640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6318154bd0 0x7f631815a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:05.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.807+0000 7f631f047640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63180c57e0 0x7f63180cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:05.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.807+0000 7f631f047640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63180c6e00 con 0x7f63180c53c0 2024-03-28T15:04:05.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.807+0000 7f631f047640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f63180c5ca0 con 0x7f63180c57e0 2024-03-28T15:04:05.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.807+0000 7f631f047640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f631815b080 con 0x7f6318154bd0 2024-03-28T15:04:05.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.808+0000 7f631d5e4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63180c57e0 0x7f63180cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:05.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.808+0000 7f6317fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f63180c53c0 0x7f63180c66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:56358/0 (socket says 172.21.15.186:56358) 2024-03-28T15:04:05.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.808+0000 7f6317fff640 1 -- 172.21.15.186:0/885356238 learned_addr learned my addr 172.21.15.186:0/885356238 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:04:05.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.808+0000 7f631cde3640 1 --2- 172.21.15.186:0/885356238 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6318154bd0 0x7f631815a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:05.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.808+0000 7f631d5e4640 1 -- 172.21.15.186:0/885356238 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6318154bd0 msgr2=0x7f631815a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:04:05.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.808+0000 7f631d5e4640 1 --2- 172.21.15.186:0/885356238 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6318154bd0 0x7f631815a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:05.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.808+0000 7f631d5e4640 1 -- 172.21.15.186:0/885356238 >> v1:172.21.15.186:6789/0 conn(0x7f63180c53c0 legacy=0x7f63180c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:05.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.808+0000 7f631d5e4640 1 -- 172.21.15.186:0/885356238 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631815b360 con 0x7f63180c57e0 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.809+0000 7f631d5e4640 1 --2- 172.21.15.186:0/885356238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63180c57e0 0x7f63180cb040 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6310009d90 tx=0x7f631002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57938b3dd10f54f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.809+0000 7f63177fe640 1 -- 172.21.15.186:0/885356238 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310039070 con 0x7f63180c57e0 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.809+0000 7f63177fe640 1 -- 172.21.15.186:0/885356238 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f631002cce0 con 0x7f63180c57e0 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.809+0000 7f63177fe640 1 -- 172.21.15.186:0/885356238 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310034660 con 0x7f63180c57e0 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.809+0000 7f631f047640 1 -- 172.21.15.186:0/885356238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63180c57e0 msgr2=0x7f63180cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.809+0000 7f631f047640 1 --2- 172.21.15.186:0/885356238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63180c57e0 0x7f63180cb040 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6310009d90 tx=0x7f631002bf30 comp rx=0 tx=0).stop 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.810+0000 7f631f047640 1 -- 172.21.15.186:0/885356238 shutdown_connections 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.810+0000 7f631f047640 1 --2- 172.21.15.186:0/885356238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63180c57e0 0x7f63180cb040 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.810+0000 7f631f047640 1 --2- 172.21.15.186:0/885356238 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6318154bd0 0x7f631815a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.810+0000 7f631f047640 1 -- 172.21.15.186:0/885356238 >> 172.21.15.186:0/885356238 conn(0x7f63180731f0 msgr2=0x7f6318073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.810+0000 7f631f047640 1 -- 172.21.15.186:0/885356238 shutdown_connections 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.810+0000 7f631f047640 1 -- 172.21.15.186:0/885356238 wait complete. 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.811+0000 7f631f047640 1 Processor -- start 2024-03-28T15:04:05.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.811+0000 7f631f047640 1 -- start start 2024-03-28T15:04:05.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f631f047640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63180c53c0 0x7f63181f3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:05.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f631f047640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63180c57e0 0x7f63180c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:05.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f631f047640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6318154bd0 0x7f63180c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:05.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f631f047640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f63180c61d0 con 0x7f63180c57e0 2024-03-28T15:04:05.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f631f047640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f63180c5ef0 con 0x7f63180c53c0 2024-03-28T15:04:05.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f631f047640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f63180cc390 con 0x7f6318154bd0 2024-03-28T15:04:05.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f6317fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63180c57e0 0x7f63180c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:05.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f6317fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63180c57e0 0x7f63180c78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:45066/0 (socket says 172.21.15.186:45066) 2024-03-28T15:04:05.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f6317fff640 1 -- 172.21.15.186:0/4209415577 learned_addr learned my addr 172.21.15.186:0/4209415577 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:04:05.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f631cde3640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63180c53c0 0x7f63181f3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:05.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f631d5e4640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6318154bd0 0x7f63180c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f6317fff640 1 -- 172.21.15.186:0/4209415577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6318154bd0 msgr2=0x7f63180c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f6317fff640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6318154bd0 0x7f63180c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f6317fff640 1 -- 172.21.15.186:0/4209415577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63180c53c0 msgr2=0x7f63181f3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f6317fff640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63180c53c0 0x7f63181f3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f6317fff640 1 -- 172.21.15.186:0/4209415577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63181f48a0 con 0x7f63180c57e0 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f631d5e4640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6318154bd0 0x7f63180c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f6317fff640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63180c57e0 0x7f63180c78f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f630400c400 tx=0x7f630400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f63157fa640 1 -- 172.21.15.186:0/4209415577 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6304017070 con 0x7f63180c57e0 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f63157fa640 1 -- 172.21.15.186:0/4209415577 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6304005ce0 con 0x7f63180c57e0 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f631f047640 1 -- 172.21.15.186:0/4209415577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63180b4ba0 con 0x7f63180c57e0 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f63157fa640 1 -- 172.21.15.186:0/4209415577 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63040072f0 con 0x7f63180c57e0 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.812+0000 7f631f047640 1 -- 172.21.15.186:0/4209415577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63180c70f0 con 0x7f63180c57e0 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.813+0000 7f631f047640 1 -- 172.21.15.186:0/4209415577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6318069500 con 0x7f63180c57e0 2024-03-28T15:04:05.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.814+0000 7f63157fa640 1 -- 172.21.15.186:0/4209415577 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f630401a070 con 0x7f63180c57e0 2024-03-28T15:04:05.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.814+0000 7f63157fa640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f62fc06c530 0x7f62fc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:05.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.815+0000 7f631cde3640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f62fc06c530 0x7f62fc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:05.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.815+0000 7f63157fa640 1 -- 172.21.15.186:0/4209415577 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7f630408ac70 con 0x7f63180c57e0 2024-03-28T15:04:05.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.815+0000 7f631cde3640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f62fc06c530 0x7f62fc06e9f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f630c006fd0 tx=0x7f630c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:05.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:05.819+0000 7f63157fa640 1 -- 172.21.15.186:0/4209415577 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f63040590e0 con 0x7f63180c57e0 2024-03-28T15:04:06.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.030+0000 7f631f047640 1 -- 172.21.15.186:0/4209415577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63180cefb0 con 0x7f63180c57e0 2024-03-28T15:04:06.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.031+0000 7f63157fa640 1 -- 172.21.15.186:0/4209415577 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6019 (secure 0 0 0) 0x7f63040589e0 con 0x7f63180c57e0 2024-03-28T15:04:06.031 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:04:06.032 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:03.487439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:04:06.032 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 17 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 -- 172.21.15.186:0/4209415577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f62fc06c530 msgr2=0x7f62fc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f62fc06c530 0x7f62fc06e9f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f630c006fd0 tx=0x7f630c008040 comp rx=0 tx=0).stop 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 -- 172.21.15.186:0/4209415577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63180c57e0 msgr2=0x7f63180c78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63180c57e0 0x7f63180c78f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f630400c400 tx=0x7f630400c8c0 comp rx=0 tx=0).stop 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 -- 172.21.15.186:0/4209415577 shutdown_connections 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f62fc06c530 0x7f62fc06e9f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6318154bd0 0x7f63180c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63180c57e0 0x7f63180c78f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 --2- 172.21.15.186:0/4209415577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63180c53c0 0x7f63181f3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 -- 172.21.15.186:0/4209415577 >> 172.21.15.186:0/4209415577 conn(0x7f63180731f0 msgr2=0x7f63181e3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 -- 172.21.15.186:0/4209415577 shutdown_connections 2024-03-28T15:04:06.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:06.033+0000 7f631f047640 1 -- 172.21.15.186:0/4209415577 wait complete. 2024-03-28T15:04:06.044 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:04:08.046 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:04:08.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.160+0000 7f09ed54b640 1 Processor -- start 2024-03-28T15:04:08.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.160+0000 7f09ed54b640 1 -- start start 2024-03-28T15:04:08.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.161+0000 7f09ed54b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e80b1e30 0x7f09e80712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:08.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.161+0000 7f09ed54b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09e80c2d00 0x7f09e80717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:08.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.161+0000 7f09ed54b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09e806caa0 con 0x7f09e8151990 2024-03-28T15:04:08.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.161+0000 7f09ed54b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f09e80c1da0 con 0x7f09e80c2d00 2024-03-28T15:04:08.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.161+0000 7f09ed54b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f09e806cd80 con 0x7f09e80b1e30 2024-03-28T15:04:08.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.162+0000 7f09e77fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09e80c2d00 0x7f09e80717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:08.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.162+0000 7f09dffff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e80b1e30 0x7f09e80712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:08.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.162+0000 7f09e77fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09e80c2d00 0x7f09e80717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:46884/0 (socket says 172.21.15.186:46884) 2024-03-28T15:04:08.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.162+0000 7f09e77fe640 1 -- 172.21.15.186:0/386679541 learned_addr learned my addr 172.21.15.186:0/386679541 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:04:08.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.162+0000 7f09dffff640 1 -- 172.21.15.186:0/386679541 >> v2:172.21.15.191:3300/0 conn(0x7f09e80c2d00 msgr2=0x7f09e80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:04:08.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.162+0000 7f09dffff640 1 --2- 172.21.15.186:0/386679541 >> v2:172.21.15.191:3300/0 conn(0x7f09e80c2d00 0x7f09e80717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:08.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.162+0000 7f09dffff640 1 -- 172.21.15.186:0/386679541 >> v1:172.21.15.191:6789/0 conn(0x7f09e8151990 legacy=0x7f09e80c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:08.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.162+0000 7f09dffff640 1 -- 172.21.15.186:0/386679541 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e80c1340 con 0x7f09e80b1e30 2024-03-28T15:04:08.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.162+0000 7f09e77fe640 1 --2- 172.21.15.186:0/386679541 >> v2:172.21.15.191:3300/0 conn(0x7f09e80c2d00 0x7f09e80717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:04:08.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.163+0000 7f09dffff640 1 --2- 172.21.15.186:0/386679541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e80b1e30 0x7f09e80712b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f09d00097a0 tx=0x7f09d002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9948af6600c64e42 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:08.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.163+0000 7f09e67fc640 1 -- 172.21.15.186:0/386679541 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d0034070 con 0x7f09e80b1e30 2024-03-28T15:04:08.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.163+0000 7f09e67fc640 1 -- 172.21.15.186:0/386679541 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09d0030610 con 0x7f09e80b1e30 2024-03-28T15:04:08.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.163+0000 7f09e67fc640 1 -- 172.21.15.186:0/386679541 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d00309e0 con 0x7f09e80b1e30 2024-03-28T15:04:08.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.164+0000 7f09ed54b640 1 -- 172.21.15.186:0/386679541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e80b1e30 msgr2=0x7f09e80712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:08.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.164+0000 7f09ed54b640 1 --2- 172.21.15.186:0/386679541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e80b1e30 0x7f09e80712b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f09d00097a0 tx=0x7f09d002b960 comp rx=0 tx=0).stop 2024-03-28T15:04:08.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.164+0000 7f09ed54b640 1 -- 172.21.15.186:0/386679541 shutdown_connections 2024-03-28T15:04:08.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.164+0000 7f09ed54b640 1 --2- 172.21.15.186:0/386679541 >> v2:172.21.15.191:3300/0 conn(0x7f09e80c2d00 0x7f09e80717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:08.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.164+0000 7f09ed54b640 1 --2- 172.21.15.186:0/386679541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e80b1e30 0x7f09e80712b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:08.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.164+0000 7f09ed54b640 1 -- 172.21.15.186:0/386679541 >> 172.21.15.186:0/386679541 conn(0x7f09e80c8a20 msgr2=0x7f09e80c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:08.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.164+0000 7f09ed54b640 1 -- 172.21.15.186:0/386679541 shutdown_connections 2024-03-28T15:04:08.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.164+0000 7f09ed54b640 1 -- 172.21.15.186:0/386679541 wait complete. 2024-03-28T15:04:08.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.165+0000 7f09ed54b640 1 Processor -- start 2024-03-28T15:04:08.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.165+0000 7f09ed54b640 1 -- start start 2024-03-28T15:04:08.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09ed54b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e80b1e30 0x7f09e8143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:08.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09ed54b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e80c2d00 0x7f09e81f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09ed54b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09e8151990 0x7f09e8162030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09ed54b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09e8071d30 con 0x7f09e80c2d00 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09ed54b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f09e80c1900 con 0x7f09e8151990 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09ed54b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f09e80c2940 con 0x7f09e80b1e30 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09dffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e80c2d00 0x7f09e81f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09dffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e80c2d00 0x7f09e81f46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:45072/0 (socket says 172.21.15.186:45072) 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09dffff640 1 -- 172.21.15.186:0/3548334306 learned_addr learned my addr 172.21.15.186:0/3548334306 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09e77fe640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09e8151990 0x7f09e8162030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09e6ffd640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e80b1e30 0x7f09e8143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09dffff640 1 -- 172.21.15.186:0/3548334306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e80b1e30 msgr2=0x7f09e8143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09dffff640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e80b1e30 0x7f09e8143d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09dffff640 1 -- 172.21.15.186:0/3548334306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09e8151990 msgr2=0x7f09e8162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09dffff640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09e8151990 0x7f09e8162030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09dffff640 1 -- 172.21.15.186:0/3548334306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e81f4bf0 con 0x7f09e80c2d00 2024-03-28T15:04:08.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09dffff640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e80c2d00 0x7f09e81f46b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f09d002bf70 tx=0x7f09d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:08.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.166+0000 7f09df7fe640 1 -- 172.21.15.186:0/3548334306 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d0034040 con 0x7f09e80c2d00 2024-03-28T15:04:08.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.167+0000 7f09df7fe640 1 -- 172.21.15.186:0/3548334306 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09d003aaa0 con 0x7f09e80c2d00 2024-03-28T15:04:08.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.167+0000 7f09ed54b640 1 -- 172.21.15.186:0/3548334306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e8163950 con 0x7f09e80c2d00 2024-03-28T15:04:08.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.167+0000 7f09df7fe640 1 -- 172.21.15.186:0/3548334306 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d00443f0 con 0x7f09e80c2d00 2024-03-28T15:04:08.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.167+0000 7f09ed54b640 1 -- 172.21.15.186:0/3548334306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e8162df0 con 0x7f09e80c2d00 2024-03-28T15:04:08.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.168+0000 7f09ed54b640 1 -- 172.21.15.186:0/3548334306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09e8069500 con 0x7f09e80c2d00 2024-03-28T15:04:08.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.168+0000 7f09df7fe640 1 -- 172.21.15.186:0/3548334306 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f09d004b060 con 0x7f09e80c2d00 2024-03-28T15:04:08.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.168+0000 7f09df7fe640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c806c530 0x7f09c806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:08.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.168+0000 7f09df7fe640 1 -- 172.21.15.186:0/3548334306 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7f09d00ad3f0 con 0x7f09e80c2d00 2024-03-28T15:04:08.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.168+0000 7f09e6ffd640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c806c530 0x7f09c806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:08.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.169+0000 7f09e6ffd640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c806c530 0x7f09c806e9f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f09d80016e0 tx=0x7f09d8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:08.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.173+0000 7f09df7fe640 1 -- 172.21.15.186:0/3548334306 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f09d003f070 con 0x7f09e80c2d00 2024-03-28T15:04:08.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.385+0000 7f09ed54b640 1 -- 172.21.15.186:0/3548334306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09e80c1080 con 0x7f09e80c2d00 2024-03-28T15:04:08.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.385+0000 7f09df7fe640 1 -- 172.21.15.186:0/3548334306 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6019 (secure 0 0 0) 0x7f09d007b330 con 0x7f09e80c2d00 2024-03-28T15:04:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:04:08.386 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:03.487439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:04:08.386 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 17 2024-03-28T15:04:08.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.387+0000 7f09ed54b640 1 -- 172.21.15.186:0/3548334306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c806c530 msgr2=0x7f09c806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:08.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.387+0000 7f09ed54b640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c806c530 0x7f09c806e9f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f09d80016e0 tx=0x7f09d8009040 comp rx=0 tx=0).stop 2024-03-28T15:04:08.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.387+0000 7f09ed54b640 1 -- 172.21.15.186:0/3548334306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e80c2d00 msgr2=0x7f09e81f46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:08.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.387+0000 7f09ed54b640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e80c2d00 0x7f09e81f46b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f09d002bf70 tx=0x7f09d00046c0 comp rx=0 tx=0).stop 2024-03-28T15:04:08.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.388+0000 7f09ed54b640 1 -- 172.21.15.186:0/3548334306 shutdown_connections 2024-03-28T15:04:08.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.388+0000 7f09ed54b640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09c806c530 0x7f09c806e9f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:08.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.388+0000 7f09ed54b640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09e8151990 0x7f09e8162030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:08.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.388+0000 7f09ed54b640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e80c2d00 0x7f09e81f46b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:08.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.388+0000 7f09ed54b640 1 --2- 172.21.15.186:0/3548334306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e80b1e30 0x7f09e8143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:08.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.388+0000 7f09ed54b640 1 -- 172.21.15.186:0/3548334306 >> 172.21.15.186:0/3548334306 conn(0x7f09e80c8a20 msgr2=0x7f09e814fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:08.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.388+0000 7f09ed54b640 1 -- 172.21.15.186:0/3548334306 shutdown_connections 2024-03-28T15:04:08.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:08.388+0000 7f09ed54b640 1 -- 172.21.15.186:0/3548334306 wait complete. 2024-03-28T15:04:08.399 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:04:10.401 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:04:11.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.518+0000 7f52f7b77640 1 Processor -- start 2024-03-28T15:04:11.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.518+0000 7f52f7b77640 1 -- start start 2024-03-28T15:04:11.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f7b77640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52f015e4d0 0x7f52f00c4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f7b77640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52f00c8eb0 con 0x7f52f0158120 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f7b77640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52f00ce270 con 0x7f52f00c30b0 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f7b77640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f52f00ce020 con 0x7f52f015e4d0 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f5913640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f52f0158120 0x7f52f0067280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:60378/0 (socket says 172.21.15.186:60378) 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f5913640 1 -- 172.21.15.186:0/3695737634 learned_addr learned my addr 172.21.15.186:0/3695737634 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f6114640 1 --2- 172.21.15.186:0/3695737634 >> v2:172.21.15.191:3300/0 conn(0x7f52f015e4d0 0x7f52f00c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f4911640 1 -- 172.21.15.186:0/3695737634 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1116500308 0 0) 0x7f52f00c8eb0 con 0x7f52f0158120 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f4911640 1 -- 172.21.15.186:0/3695737634 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52dc003520 con 0x7f52f0158120 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f4911640 1 -- 172.21.15.186:0/3695737634 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3492889810 0 0) 0x7f52dc003520 con 0x7f52f0158120 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f6114640 1 -- 172.21.15.186:0/3695737634 >> v1:172.21.15.191:6789/0 conn(0x7f52f00c30b0 legacy=0x7f52f015ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f6114640 1 -- 172.21.15.186:0/3695737634 >> v1:172.21.15.186:6789/0 conn(0x7f52f0158120 legacy=0x7f52f0067280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:11.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.519+0000 7f52f6114640 1 -- 172.21.15.186:0/3695737634 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f00ce550 con 0x7f52f015e4d0 2024-03-28T15:04:11.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.520+0000 7f52f6114640 1 --2- 172.21.15.186:0/3695737634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52f015e4d0 0x7f52f00c4440 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009d90 tx=0x7f52e802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33063d1ef1153f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:11.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.520+0000 7f52f4911640 1 -- 172.21.15.186:0/3695737634 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8039070 con 0x7f52f015e4d0 2024-03-28T15:04:11.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.520+0000 7f52f4911640 1 -- 172.21.15.186:0/3695737634 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e802c8b0 con 0x7f52f015e4d0 2024-03-28T15:04:11.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.520+0000 7f52f4911640 1 -- 172.21.15.186:0/3695737634 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ec001670 con 0x7f52f015e4d0 2024-03-28T15:04:11.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.521+0000 7f52f7b77640 1 -- 172.21.15.186:0/3695737634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52f015e4d0 msgr2=0x7f52f00c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:11.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.521+0000 7f52f7b77640 1 --2- 172.21.15.186:0/3695737634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52f015e4d0 0x7f52f00c4440 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009d90 tx=0x7f52e802bd80 comp rx=0 tx=0).stop 2024-03-28T15:04:11.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.521+0000 7f52f7b77640 1 -- 172.21.15.186:0/3695737634 shutdown_connections 2024-03-28T15:04:11.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.521+0000 7f52f7b77640 1 --2- 172.21.15.186:0/3695737634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52f015e4d0 0x7f52f00c4440 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:11.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.521+0000 7f52f7b77640 1 -- 172.21.15.186:0/3695737634 >> 172.21.15.186:0/3695737634 conn(0x7f52f006fc80 msgr2=0x7f52f00734c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:11.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.521+0000 7f52f7b77640 1 -- 172.21.15.186:0/3695737634 shutdown_connections 2024-03-28T15:04:11.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.521+0000 7f52f7b77640 1 -- 172.21.15.186:0/3695737634 wait complete. 2024-03-28T15:04:11.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.522+0000 7f52f7b77640 1 Processor -- start 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.522+0000 7f52f7b77640 1 -- start start 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f7b77640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52f00c30b0 0x7f52f0155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f7b77640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52f0158120 0x7f52f015d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f7b77640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52f015e4d0 0x7f52f015d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f7b77640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52f00c5a80 con 0x7f52f0158120 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f7b77640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f52f00ceb10 con 0x7f52f015e4d0 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f7b77640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f52f00c8a90 con 0x7f52f00c30b0 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f5112640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52f0158120 0x7f52f015d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f5112640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52f0158120 0x7f52f015d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39364/0 (socket says 172.21.15.186:39364) 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f5112640 1 -- 172.21.15.186:0/3478098716 learned_addr learned my addr 172.21.15.186:0/3478098716 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f6114640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52f015e4d0 0x7f52f015d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f5913640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52f00c30b0 0x7f52f0155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f5112640 1 -- 172.21.15.186:0/3478098716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52f00c30b0 msgr2=0x7f52f0155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:11.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f5112640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52f00c30b0 0x7f52f0155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:11.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f5112640 1 -- 172.21.15.186:0/3478098716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52f015e4d0 msgr2=0x7f52f015d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:11.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f5112640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52f015e4d0 0x7f52f015d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:11.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f5112640 1 -- 172.21.15.186:0/3478098716 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f015b260 con 0x7f52f0158120 2024-03-28T15:04:11.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f5913640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52f00c30b0 0x7f52f0155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:04:11.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.523+0000 7f52f6114640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52f015e4d0 0x7f52f015d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:04:11.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.524+0000 7f52f5112640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52f0158120 0x7f52f015d290 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f52e000c400 tx=0x7f52e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:11.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.524+0000 7f52e67fc640 1 -- 172.21.15.186:0/3478098716 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0017070 con 0x7f52f0158120 2024-03-28T15:04:11.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.524+0000 7f52e67fc640 1 -- 172.21.15.186:0/3478098716 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e00052d0 con 0x7f52f0158120 2024-03-28T15:04:11.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.524+0000 7f52f7b77640 1 -- 172.21.15.186:0/3478098716 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f0159240 con 0x7f52f0158120 2024-03-28T15:04:11.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.524+0000 7f52e67fc640 1 -- 172.21.15.186:0/3478098716 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0003830 con 0x7f52f0158120 2024-03-28T15:04:11.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.524+0000 7f52f7b77640 1 -- 172.21.15.186:0/3478098716 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f0159800 con 0x7f52f0158120 2024-03-28T15:04:11.531 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.525+0000 7f52f7b77640 1 -- 172.21.15.186:0/3478098716 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52f00c5490 con 0x7f52f0158120 2024-03-28T15:04:11.531 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.525+0000 7f52e67fc640 1 -- 172.21.15.186:0/3478098716 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f52e001a070 con 0x7f52f0158120 2024-03-28T15:04:11.531 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.526+0000 7f52e67fc640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52d406c530 0x7f52d406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:11.531 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.526+0000 7f52f5913640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52d406c530 0x7f52d406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:11.531 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.526+0000 7f52e67fc640 1 -- 172.21.15.186:0/3478098716 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7f52e008ac10 con 0x7f52f0158120 2024-03-28T15:04:11.531 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.526+0000 7f52f5913640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52d406c530 0x7f52d406e9f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0011e0 tx=0x7f52ec009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:11.531 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.530+0000 7f52e67fc640 1 -- 172.21.15.186:0/3478098716 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f52f00c5490 con 0x7f52f0158120 2024-03-28T15:04:11.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.744+0000 7f52f7b77640 1 -- 172.21.15.186:0/3478098716 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52f00cde00 con 0x7f52f0158120 2024-03-28T15:04:11.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.744+0000 7f52e67fc640 1 -- 172.21.15.186:0/3478098716 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6020 (secure 0 0 0) 0x7f52f00cde00 con 0x7f52f0158120 2024-03-28T15:04:11.745 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:04:11.745 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:04:11.745 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 18 2024-03-28T15:04:11.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 -- 172.21.15.186:0/3478098716 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52d406c530 msgr2=0x7f52d406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:11.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52d406c530 0x7f52d406e9f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0011e0 tx=0x7f52ec009040 comp rx=0 tx=0).stop 2024-03-28T15:04:11.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 -- 172.21.15.186:0/3478098716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52f0158120 msgr2=0x7f52f015d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:11.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52f0158120 0x7f52f015d290 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f52e000c400 tx=0x7f52e000c8c0 comp rx=0 tx=0).stop 2024-03-28T15:04:11.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 -- 172.21.15.186:0/3478098716 shutdown_connections 2024-03-28T15:04:11.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f52d406c530 0x7f52d406e9f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:11.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52f015e4d0 0x7f52f015d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:11.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52f0158120 0x7f52f015d290 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:11.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 --2- 172.21.15.186:0/3478098716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52f00c30b0 0x7f52f0155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:11.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 -- 172.21.15.186:0/3478098716 >> 172.21.15.186:0/3478098716 conn(0x7f52f006fc80 msgr2=0x7f52f0070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:11.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 -- 172.21.15.186:0/3478098716 shutdown_connections 2024-03-28T15:04:11.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:11.746+0000 7f52f7b77640 1 -- 172.21.15.186:0/3478098716 wait complete. 2024-03-28T15:04:11.758 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=4302 2024-03-28T15:04:11.758 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.b 2024-03-28T15:04:13.759 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-03-28T15:04:13.759 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T15:04:13.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T15:04:13.763 INFO:tasks.ceph.mds.b:Started 2024-03-28T15:04:13.763 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:04:13.846 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2024-03-28T15:04:13.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.874+0000 7f100bc88640 1 Processor -- start 2024-03-28T15:04:13.874 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.874+0000 7f100bc88640 1 -- start start 2024-03-28T15:04:13.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.875+0000 7f100bc88640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1004151990 0x7f10040c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:13.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.875+0000 7f100bc88640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10040c2d00 0x7f10040c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:13.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.875+0000 7f100bc88640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100406caa0 con 0x7f10040b1e30 2024-03-28T15:04:13.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.875+0000 7f100bc88640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f10040c1da0 con 0x7f10040c2d00 2024-03-28T15:04:13.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.875+0000 7f100bc88640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f100406cd80 con 0x7f1004151990 2024-03-28T15:04:13.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.875+0000 7f100a225640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10040c2d00 0x7f10040c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:13.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.875+0000 7f1009223640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f10040b1e30 0x7f10040712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:60382/0 (socket says 172.21.15.186:60382) 2024-03-28T15:04:13.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.875+0000 7f100a225640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10040c2d00 0x7f10040c2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39370/0 (socket says 172.21.15.186:39370) 2024-03-28T15:04:13.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.875+0000 7f1009223640 1 -- 172.21.15.186:0/2133089553 learned_addr learned my addr 172.21.15.186:0/2133089553 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:04:13.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.875+0000 7f1009a24640 1 --2- 172.21.15.186:0/2133089553 >> v2:172.21.15.191:3300/0 conn(0x7f1004151990 0x7f10040c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:13.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.876+0000 7f100a225640 1 -- 172.21.15.186:0/2133089553 >> v2:172.21.15.191:3300/0 conn(0x7f1004151990 msgr2=0x7f10040c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:04:13.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.876+0000 7f100a225640 1 --2- 172.21.15.186:0/2133089553 >> v2:172.21.15.191:3300/0 conn(0x7f1004151990 0x7f10040c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:13.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.876+0000 7f100a225640 1 -- 172.21.15.186:0/2133089553 >> v1:172.21.15.186:6789/0 conn(0x7f10040b1e30 legacy=0x7f10040712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:13.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.876+0000 7f100a225640 1 -- 172.21.15.186:0/2133089553 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10040c1340 con 0x7f10040c2d00 2024-03-28T15:04:13.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.876+0000 7f1008a22640 1 -- 172.21.15.186:0/2133089553 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3957734727 0 0) 0x7f100406caa0 con 0x7f10040b1e30 2024-03-28T15:04:13.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.876+0000 7f1009a24640 1 --2- 172.21.15.186:0/2133089553 >> v2:172.21.15.191:3300/0 conn(0x7f1004151990 0x7f10040c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:04:13.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.876+0000 7f100a225640 1 --2- 172.21.15.186:0/2133089553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10040c2d00 0x7f10040c2770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f100406fd90 tx=0x7f0ffc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3602ae9ffc2314f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:13.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.876+0000 7f1008a22640 1 -- 172.21.15.186:0/2133089553 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc034070 con 0x7f10040c2d00 2024-03-28T15:04:13.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.876+0000 7f1008a22640 1 -- 172.21.15.186:0/2133089553 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ffc030610 con 0x7f10040c2d00 2024-03-28T15:04:13.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.876+0000 7f1008a22640 1 -- 172.21.15.186:0/2133089553 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc0309e0 con 0x7f10040c2d00 2024-03-28T15:04:13.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.877+0000 7f100bc88640 1 -- 172.21.15.186:0/2133089553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10040c2d00 msgr2=0x7f10040c2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:13.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.877+0000 7f100bc88640 1 --2- 172.21.15.186:0/2133089553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10040c2d00 0x7f10040c2770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f100406fd90 tx=0x7f0ffc02b960 comp rx=0 tx=0).stop 2024-03-28T15:04:13.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.877+0000 7f100bc88640 1 -- 172.21.15.186:0/2133089553 shutdown_connections 2024-03-28T15:04:13.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.877+0000 7f100bc88640 1 --2- 172.21.15.186:0/2133089553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10040c2d00 0x7f10040c2770 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:13.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.877+0000 7f100bc88640 1 --2- 172.21.15.186:0/2133089553 >> v2:172.21.15.191:3300/0 conn(0x7f1004151990 0x7f10040c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:13.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.877+0000 7f100bc88640 1 -- 172.21.15.186:0/2133089553 >> 172.21.15.186:0/2133089553 conn(0x7f10040c8a20 msgr2=0x7f10040c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:13.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.877+0000 7f100bc88640 1 -- 172.21.15.186:0/2133089553 shutdown_connections 2024-03-28T15:04:13.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.877+0000 7f100bc88640 1 -- 172.21.15.186:0/2133089553 wait complete. 2024-03-28T15:04:13.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.878+0000 7f100bc88640 1 Processor -- start 2024-03-28T15:04:13.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.878+0000 7f100bc88640 1 -- start start 2024-03-28T15:04:13.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100bc88640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10040b1e30 0x7f10040cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:13.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100bc88640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10040c2d00 0x7f10041e2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:13.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100bc88640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1004151990 0x7f10041e32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:13.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100bc88640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1004073dd0 con 0x7f1004151990 2024-03-28T15:04:13.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100bc88640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f10040c1900 con 0x7f10040b1e30 2024-03-28T15:04:13.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100bc88640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f1004073f50 con 0x7f10040c2d00 2024-03-28T15:04:13.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100a225640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1004151990 0x7f10041e32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:13.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100a225640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1004151990 0x7f10041e32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39378/0 (socket says 172.21.15.186:39378) 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100a225640 1 -- 172.21.15.186:0/1452100858 learned_addr learned my addr 172.21.15.186:0/1452100858 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f1009223640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10040c2d00 0x7f10041e2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f1009a24640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10040b1e30 0x7f10040cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100a225640 1 -- 172.21.15.186:0/1452100858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10040c2d00 msgr2=0x7f10041e2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100a225640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10040c2d00 0x7f10041e2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100a225640 1 -- 172.21.15.186:0/1452100858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10040b1e30 msgr2=0x7f10040cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100a225640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10040b1e30 0x7f10040cafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.879+0000 7f100a225640 1 -- 172.21.15.186:0/1452100858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1004155cc0 con 0x7f1004151990 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.880+0000 7f100a225640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1004151990 0x7f10041e32e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc02bf70 tx=0x7f0ffc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.880+0000 7f0ffa7fc640 1 -- 172.21.15.186:0/1452100858 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc034040 con 0x7f1004151990 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.880+0000 7f0ffa7fc640 1 -- 172.21.15.186:0/1452100858 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ffc03ac40 con 0x7f1004151990 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.880+0000 7f100bc88640 1 -- 172.21.15.186:0/1452100858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1004155420 con 0x7f1004151990 2024-03-28T15:04:13.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.880+0000 7f0ffa7fc640 1 -- 172.21.15.186:0/1452100858 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc006e10 con 0x7f1004151990 2024-03-28T15:04:13.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.880+0000 7f100bc88640 1 -- 172.21.15.186:0/1452100858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10041559e0 con 0x7f1004151990 2024-03-28T15:04:13.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.881+0000 7f100bc88640 1 -- 172.21.15.186:0/1452100858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1004069500 con 0x7f1004151990 2024-03-28T15:04:13.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.881+0000 7f0ffa7fc640 1 -- 172.21.15.186:0/1452100858 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0ffc037070 con 0x7f1004151990 2024-03-28T15:04:13.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.882+0000 7f0ffa7fc640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0fe806c560 0x7f0fe806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:13.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.882+0000 7f0ffa7fc640 1 -- 172.21.15.186:0/1452100858 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7f0ffc0ad540 con 0x7f1004151990 2024-03-28T15:04:13.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.882+0000 7f1009a24640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0fe806c560 0x7f0fe806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:13.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.882+0000 7f1009a24640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0fe806c560 0x7f0fe806ea20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1000003b80 tx=0x7f1000009910 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:13.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:13.886+0000 7f0ffa7fc640 1 -- 172.21.15.186:0/1452100858 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f0ffc03f070 con 0x7f1004151990 2024-03-28T15:04:14.098 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.098+0000 7f100bc88640 1 -- 172.21.15.186:0/1452100858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10040b1a00 con 0x7f1004151990 2024-03-28T15:04:14.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.098+0000 7f0ffa7fc640 1 -- 172.21.15.186:0/1452100858 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6020 (secure 0 0 0) 0x7f0ffc07b480 con 0x7f1004151990 2024-03-28T15:04:14.100 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:04:14.100 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:04:14.100 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 18 2024-03-28T15:04:14.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.100+0000 7f100bc88640 1 -- 172.21.15.186:0/1452100858 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0fe806c560 msgr2=0x7f0fe806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:14.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.100+0000 7f100bc88640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0fe806c560 0x7f0fe806ea20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1000003b80 tx=0x7f1000009910 comp rx=0 tx=0).stop 2024-03-28T15:04:14.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.100+0000 7f100bc88640 1 -- 172.21.15.186:0/1452100858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1004151990 msgr2=0x7f10041e32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:14.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.100+0000 7f100bc88640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1004151990 0x7f10041e32e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc02bf70 tx=0x7f0ffc0046c0 comp rx=0 tx=0).stop 2024-03-28T15:04:14.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.100+0000 7f100bc88640 1 -- 172.21.15.186:0/1452100858 shutdown_connections 2024-03-28T15:04:14.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.100+0000 7f100bc88640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0fe806c560 0x7f0fe806ea20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:14.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.100+0000 7f100bc88640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1004151990 0x7f10041e32e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:14.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.100+0000 7f100bc88640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10040c2d00 0x7f10041e2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:14.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.100+0000 7f100bc88640 1 --2- 172.21.15.186:0/1452100858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10040b1e30 0x7f10040cafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:14.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.100+0000 7f100bc88640 1 -- 172.21.15.186:0/1452100858 >> 172.21.15.186:0/1452100858 conn(0x7f10040c8a20 msgr2=0x7f100414f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:14.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.101+0000 7f100bc88640 1 -- 172.21.15.186:0/1452100858 shutdown_connections 2024-03-28T15:04:14.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:14.101+0000 7f100bc88640 1 -- 172.21.15.186:0/1452100858 wait complete. 2024-03-28T15:04:14.111 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-03-28T15:04:15.808 INFO:tasks.workunit.client.0.smithi186.stdout: create 218/30 file after the snapshot 2024-03-28T15:04:15.811 INFO:tasks.workunit.client.0.smithi186.stdout: create 218/31 file after the snapshot 2024-03-28T15:04:15.813 INFO:tasks.workunit.client.0.smithi186.stdout: create 218/32 file after the snapshot 2024-03-28T15:04:15.815 INFO:tasks.workunit.client.0.smithi186.stdout: create 218/33 file after the snapshot 2024-03-28T15:04:15.818 INFO:tasks.workunit.client.0.smithi186.stdout: create 218/34 file after the snapshot 2024-03-28T15:04:15.821 INFO:tasks.workunit.client.0.smithi186.stdout: create 218/35 file after the snapshot 2024-03-28T15:04:15.827 INFO:tasks.workunit.client.0.smithi186.stdout: create 218/36 file after the snapshot 2024-03-28T15:04:15.827 INFO:tasks.workunit.client.0.smithi186.stdout: create 218/37 file after the snapshot 2024-03-28T15:04:15.828 INFO:tasks.workunit.client.0.smithi186.stdout: create 218/38 file after the snapshot 2024-03-28T15:04:15.830 INFO:tasks.workunit.client.0.smithi186.stdout: create 218/39 file after the snapshot 2024-03-28T15:04:15.833 INFO:tasks.workunit.client.0.smithi186.stdout: create 219/.snap/snap-subdir-test 2024-03-28T15:04:15.849 INFO:tasks.workunit.client.0.smithi186.stdout: create 219/30 file after the snapshot 2024-03-28T15:04:15.851 INFO:tasks.workunit.client.0.smithi186.stdout: create 219/31 file after the snapshot 2024-03-28T15:04:15.853 INFO:tasks.workunit.client.0.smithi186.stdout: create 219/32 file after the snapshot 2024-03-28T15:04:15.856 INFO:tasks.workunit.client.0.smithi186.stdout: create 219/33 file after the snapshot 2024-03-28T15:04:15.858 INFO:tasks.workunit.client.0.smithi186.stdout: create 219/34 file after the snapshot 2024-03-28T15:04:15.860 INFO:tasks.workunit.client.0.smithi186.stdout: create 219/35 file after the snapshot 2024-03-28T15:04:15.862 INFO:tasks.workunit.client.0.smithi186.stdout: create 219/36 file after the snapshot 2024-03-28T15:04:15.864 INFO:tasks.workunit.client.0.smithi186.stdout: create 219/37 file after the snapshot 2024-03-28T15:04:15.867 INFO:tasks.workunit.client.0.smithi186.stdout: create 219/38 file after the snapshot 2024-03-28T15:04:15.869 INFO:tasks.workunit.client.0.smithi186.stdout: create 219/39 file after the snapshot 2024-03-28T15:04:15.871 INFO:tasks.workunit.client.0.smithi186.stdout: create 220/.snap/snap-subdir-test 2024-03-28T15:04:15.887 INFO:tasks.workunit.client.0.smithi186.stdout: create 220/30 file after the snapshot 2024-03-28T15:04:15.889 INFO:tasks.workunit.client.0.smithi186.stdout: create 220/31 file after the snapshot 2024-03-28T15:04:15.891 INFO:tasks.workunit.client.0.smithi186.stdout: create 220/32 file after the snapshot 2024-03-28T15:04:15.894 INFO:tasks.workunit.client.0.smithi186.stdout: create 220/33 file after the snapshot 2024-03-28T15:04:15.897 INFO:tasks.workunit.client.0.smithi186.stdout: create 220/34 file after the snapshot 2024-03-28T15:04:15.900 INFO:tasks.workunit.client.0.smithi186.stdout: create 220/35 file after the snapshot 2024-03-28T15:04:15.902 INFO:tasks.workunit.client.0.smithi186.stdout: create 220/36 file after the snapshot 2024-03-28T15:04:15.906 INFO:tasks.workunit.client.0.smithi186.stdout: create 220/37 file after the snapshot 2024-03-28T15:04:15.907 INFO:tasks.workunit.client.0.smithi186.stdout: create 220/38 file after the snapshot 2024-03-28T15:04:15.909 INFO:tasks.workunit.client.0.smithi186.stdout: create 220/39 file after the snapshot 2024-03-28T15:04:15.911 INFO:tasks.workunit.client.0.smithi186.stdout: create 221/.snap/snap-subdir-test 2024-03-28T15:04:15.923 INFO:tasks.workunit.client.0.smithi186.stdout: create 221/30 file after the snapshot 2024-03-28T15:04:15.925 INFO:tasks.workunit.client.0.smithi186.stdout: create 221/31 file after the snapshot 2024-03-28T15:04:15.927 INFO:tasks.workunit.client.0.smithi186.stdout: create 221/32 file after the snapshot 2024-03-28T15:04:15.929 INFO:tasks.workunit.client.0.smithi186.stdout: create 221/33 file after the snapshot 2024-03-28T15:04:15.931 INFO:tasks.workunit.client.0.smithi186.stdout: create 221/34 file after the snapshot 2024-03-28T15:04:15.934 INFO:tasks.workunit.client.0.smithi186.stdout: create 221/35 file after the snapshot 2024-03-28T15:04:15.936 INFO:tasks.workunit.client.0.smithi186.stdout: create 221/36 file after the snapshot 2024-03-28T15:04:15.938 INFO:tasks.workunit.client.0.smithi186.stdout: create 221/37 file after the snapshot 2024-03-28T15:04:15.941 INFO:tasks.workunit.client.0.smithi186.stdout: create 221/38 file after the snapshot 2024-03-28T15:04:15.943 INFO:tasks.workunit.client.0.smithi186.stdout: create 221/39 file after the snapshot 2024-03-28T15:04:15.946 INFO:tasks.workunit.client.0.smithi186.stdout: create 222/.snap/snap-subdir-test 2024-03-28T15:04:15.962 INFO:tasks.workunit.client.0.smithi186.stdout: create 222/30 file after the snapshot 2024-03-28T15:04:15.964 INFO:tasks.workunit.client.0.smithi186.stdout: create 222/31 file after the snapshot 2024-03-28T15:04:15.967 INFO:tasks.workunit.client.0.smithi186.stdout: create 222/32 file after the snapshot 2024-03-28T15:04:15.969 INFO:tasks.workunit.client.0.smithi186.stdout: create 222/33 file after the snapshot 2024-03-28T15:04:15.972 INFO:tasks.workunit.client.0.smithi186.stdout: create 222/34 file after the snapshot 2024-03-28T15:04:15.975 INFO:tasks.workunit.client.0.smithi186.stdout: create 222/35 file after the snapshot 2024-03-28T15:04:15.977 INFO:tasks.workunit.client.0.smithi186.stdout: create 222/36 file after the snapshot 2024-03-28T15:04:15.979 INFO:tasks.workunit.client.0.smithi186.stdout: create 222/37 file after the snapshot 2024-03-28T15:04:15.982 INFO:tasks.workunit.client.0.smithi186.stdout: create 222/38 file after the snapshot 2024-03-28T15:04:15.984 INFO:tasks.workunit.client.0.smithi186.stdout: create 222/39 file after the snapshot 2024-03-28T15:04:15.987 INFO:tasks.workunit.client.0.smithi186.stdout: create 223/.snap/snap-subdir-test 2024-03-28T15:04:15.999 INFO:tasks.workunit.client.0.smithi186.stdout: create 223/30 file after the snapshot 2024-03-28T15:04:16.001 INFO:tasks.workunit.client.0.smithi186.stdout: create 223/31 file after the snapshot 2024-03-28T15:04:16.003 INFO:tasks.workunit.client.0.smithi186.stdout: create 223/32 file after the snapshot 2024-03-28T15:04:16.005 INFO:tasks.workunit.client.0.smithi186.stdout: create 223/33 file after the snapshot 2024-03-28T15:04:16.007 INFO:tasks.workunit.client.0.smithi186.stdout: create 223/34 file after the snapshot 2024-03-28T15:04:16.009 INFO:tasks.workunit.client.0.smithi186.stdout: create 223/35 file after the snapshot 2024-03-28T15:04:16.012 INFO:tasks.workunit.client.0.smithi186.stdout: create 223/36 file after the snapshot 2024-03-28T15:04:16.014 INFO:tasks.workunit.client.0.smithi186.stdout: create 223/37 file after the snapshot 2024-03-28T15:04:16.017 INFO:tasks.workunit.client.0.smithi186.stdout: create 223/38 file after the snapshot 2024-03-28T15:04:16.018 INFO:tasks.workunit.client.0.smithi186.stdout: create 223/39 file after the snapshot 2024-03-28T15:04:16.021 INFO:tasks.workunit.client.0.smithi186.stdout: create 224/.snap/snap-subdir-test 2024-03-28T15:04:16.036 INFO:tasks.workunit.client.0.smithi186.stdout: create 224/30 file after the snapshot 2024-03-28T15:04:16.038 INFO:tasks.workunit.client.0.smithi186.stdout: create 224/31 file after the snapshot 2024-03-28T15:04:16.041 INFO:tasks.workunit.client.0.smithi186.stdout: create 224/32 file after the snapshot 2024-03-28T15:04:16.042 INFO:tasks.workunit.client.0.smithi186.stdout: create 224/33 file after the snapshot 2024-03-28T15:04:16.044 INFO:tasks.workunit.client.0.smithi186.stdout: create 224/34 file after the snapshot 2024-03-28T15:04:16.046 INFO:tasks.workunit.client.0.smithi186.stdout: create 224/35 file after the snapshot 2024-03-28T15:04:16.049 INFO:tasks.workunit.client.0.smithi186.stdout: create 224/36 file after the snapshot 2024-03-28T15:04:16.051 INFO:tasks.workunit.client.0.smithi186.stdout: create 224/37 file after the snapshot 2024-03-28T15:04:16.053 INFO:tasks.workunit.client.0.smithi186.stdout: create 224/38 file after the snapshot 2024-03-28T15:04:16.055 INFO:tasks.workunit.client.0.smithi186.stdout: create 224/39 file after the snapshot 2024-03-28T15:04:16.057 INFO:tasks.workunit.client.0.smithi186.stdout: create 225/.snap/snap-subdir-test 2024-03-28T15:04:16.072 INFO:tasks.workunit.client.0.smithi186.stdout: create 225/30 file after the snapshot 2024-03-28T15:04:16.074 INFO:tasks.workunit.client.0.smithi186.stdout: create 225/31 file after the snapshot 2024-03-28T15:04:16.076 INFO:tasks.workunit.client.0.smithi186.stdout: create 225/32 file after the snapshot 2024-03-28T15:04:16.079 INFO:tasks.workunit.client.0.smithi186.stdout: create 225/33 file after the snapshot 2024-03-28T15:04:16.081 INFO:tasks.workunit.client.0.smithi186.stdout: create 225/34 file after the snapshot 2024-03-28T15:04:16.084 INFO:tasks.workunit.client.0.smithi186.stdout: create 225/35 file after the snapshot 2024-03-28T15:04:16.086 INFO:tasks.workunit.client.0.smithi186.stdout: create 225/36 file after the snapshot 2024-03-28T15:04:16.088 INFO:tasks.workunit.client.0.smithi186.stdout: create 225/37 file after the snapshot 2024-03-28T15:04:16.090 INFO:tasks.workunit.client.0.smithi186.stdout: create 225/38 file after the snapshot 2024-03-28T15:04:16.092 INFO:tasks.workunit.client.0.smithi186.stdout: create 225/39 file after the snapshot 2024-03-28T15:04:16.094 INFO:tasks.workunit.client.0.smithi186.stdout: create 226/.snap/snap-subdir-test 2024-03-28T15:04:16.109 INFO:tasks.workunit.client.0.smithi186.stdout: create 226/30 file after the snapshot 2024-03-28T15:04:16.111 INFO:tasks.workunit.client.0.smithi186.stdout: create 226/31 file after the snapshot 2024-03-28T15:04:16.113 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:04:16.114 INFO:tasks.workunit.client.0.smithi186.stdout: create 226/32 file after the snapshot 2024-03-28T15:04:16.117 INFO:tasks.workunit.client.0.smithi186.stdout: create 226/33 file after the snapshot 2024-03-28T15:04:16.117 INFO:tasks.workunit.client.0.smithi186.stdout: create 226/34 file after the snapshot 2024-03-28T15:04:16.119 INFO:tasks.workunit.client.0.smithi186.stdout: create 226/35 file after the snapshot 2024-03-28T15:04:16.122 INFO:tasks.workunit.client.0.smithi186.stdout: create 226/36 file after the snapshot 2024-03-28T15:04:16.124 INFO:tasks.workunit.client.0.smithi186.stdout: create 226/37 file after the snapshot 2024-03-28T15:04:16.127 INFO:tasks.workunit.client.0.smithi186.stdout: create 226/38 file after the snapshot 2024-03-28T15:04:16.129 INFO:tasks.workunit.client.0.smithi186.stdout: create 226/39 file after the snapshot 2024-03-28T15:04:16.132 INFO:tasks.workunit.client.0.smithi186.stdout: create 227/.snap/snap-subdir-test 2024-03-28T15:04:16.147 INFO:tasks.workunit.client.0.smithi186.stdout: create 227/30 file after the snapshot 2024-03-28T15:04:16.149 INFO:tasks.workunit.client.0.smithi186.stdout: create 227/31 file after the snapshot 2024-03-28T15:04:16.151 INFO:tasks.workunit.client.0.smithi186.stdout: create 227/32 file after the snapshot 2024-03-28T15:04:16.153 INFO:tasks.workunit.client.0.smithi186.stdout: create 227/33 file after the snapshot 2024-03-28T15:04:16.156 INFO:tasks.workunit.client.0.smithi186.stdout: create 227/34 file after the snapshot 2024-03-28T15:04:16.158 INFO:tasks.workunit.client.0.smithi186.stdout: create 227/35 file after the snapshot 2024-03-28T15:04:16.161 INFO:tasks.workunit.client.0.smithi186.stdout: create 227/36 file after the snapshot 2024-03-28T15:04:16.163 INFO:tasks.workunit.client.0.smithi186.stdout: create 227/37 file after the snapshot 2024-03-28T15:04:16.165 INFO:tasks.workunit.client.0.smithi186.stdout: create 227/38 file after the snapshot 2024-03-28T15:04:16.168 INFO:tasks.workunit.client.0.smithi186.stdout: create 227/39 file after the snapshot 2024-03-28T15:04:16.170 INFO:tasks.workunit.client.0.smithi186.stdout: create 228/.snap/snap-subdir-test 2024-03-28T15:04:16.184 INFO:tasks.workunit.client.0.smithi186.stdout: create 228/30 file after the snapshot 2024-03-28T15:04:16.186 INFO:tasks.workunit.client.0.smithi186.stdout: create 228/31 file after the snapshot 2024-03-28T15:04:16.189 INFO:tasks.workunit.client.0.smithi186.stdout: create 228/32 file after the snapshot 2024-03-28T15:04:16.192 INFO:tasks.workunit.client.0.smithi186.stdout: create 228/33 file after the snapshot 2024-03-28T15:04:16.194 INFO:tasks.workunit.client.0.smithi186.stdout: create 228/34 file after the snapshot 2024-03-28T15:04:16.197 INFO:tasks.workunit.client.0.smithi186.stdout: create 228/35 file after the snapshot 2024-03-28T15:04:16.199 INFO:tasks.workunit.client.0.smithi186.stdout: create 228/36 file after the snapshot 2024-03-28T15:04:16.201 INFO:tasks.workunit.client.0.smithi186.stdout: create 228/37 file after the snapshot 2024-03-28T15:04:16.202 INFO:tasks.workunit.client.0.smithi186.stdout: create 228/38 file after the snapshot 2024-03-28T15:04:16.205 INFO:tasks.workunit.client.0.smithi186.stdout: create 228/39 file after the snapshot 2024-03-28T15:04:16.208 INFO:tasks.workunit.client.0.smithi186.stdout: create 229/.snap/snap-subdir-test 2024-03-28T15:04:16.220 INFO:tasks.workunit.client.0.smithi186.stdout: create 229/30 file after the snapshot 2024-03-28T15:04:16.222 INFO:tasks.workunit.client.0.smithi186.stdout: create 229/31 file after the snapshot 2024-03-28T15:04:16.224 INFO:tasks.workunit.client.0.smithi186.stdout: create 229/32 file after the snapshot 2024-03-28T15:04:16.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.226+0000 7f6333fbc640 1 Processor -- start 2024-03-28T15:04:16.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.226+0000 7f6333fbc640 1 -- start start 2024-03-28T15:04:16.226 INFO:tasks.workunit.client.0.smithi186.stdout: create 229/33 file after the snapshot 2024-03-28T15:04:16.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.227+0000 7f6333fbc640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f632c0b1eb0 0x7f632c06b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:16.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.227+0000 7f6333fbc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f632c0c7a80 0x7f632c06bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:16.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.227+0000 7f6333fbc640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f632c0b2290 con 0x7f632c06c490 2024-03-28T15:04:16.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.227+0000 7f6333fbc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f632c15b670 con 0x7f632c0c7a80 2024-03-28T15:04:16.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.227+0000 7f6333fbc640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f632c15aa80 con 0x7f632c0b1eb0 2024-03-28T15:04:16.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.227+0000 7f6332559640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f632c0c7a80 0x7f632c06bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:16.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.227+0000 7f6331d58640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f632c0b1eb0 0x7f632c06b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:16.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.227+0000 7f6332559640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f632c0c7a80 0x7f632c06bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39388/0 (socket says 172.21.15.186:39388) 2024-03-28T15:04:16.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.227+0000 7f6331557640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f632c06c490 0x7f632c15dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:56454/0 (socket says 172.21.15.186:56454) 2024-03-28T15:04:16.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.227+0000 7f6332559640 1 -- 172.21.15.186:0/3317090084 learned_addr learned my addr 172.21.15.186:0/3317090084 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:04:16.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.228+0000 7f6332559640 1 -- 172.21.15.186:0/3317090084 >> v2:172.21.15.191:3300/0 conn(0x7f632c0b1eb0 msgr2=0x7f632c06b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:04:16.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.228+0000 7f6332559640 1 --2- 172.21.15.186:0/3317090084 >> v2:172.21.15.191:3300/0 conn(0x7f632c0b1eb0 0x7f632c06b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:16.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.228+0000 7f6332559640 1 -- 172.21.15.186:0/3317090084 >> v1:172.21.15.191:6789/0 conn(0x7f632c06c490 legacy=0x7f632c15dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:16.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.228+0000 7f6330d56640 1 -- 172.21.15.186:0/3317090084 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 223655887 0 0) 0x7f632c0b2290 con 0x7f632c06c490 2024-03-28T15:04:16.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.228+0000 7f6332559640 1 -- 172.21.15.186:0/3317090084 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632c0c98d0 con 0x7f632c0c7a80 2024-03-28T15:04:16.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.228+0000 7f6332559640 1 --2- 172.21.15.186:0/3317090084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c0c7a80 0x7f632c06bc50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f632c070c20 tx=0x7f632402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451dd00b3c1511ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:16.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.229+0000 7f6330d56640 1 -- 172.21.15.186:0/3317090084 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632402cbb0 con 0x7f632c0c7a80 2024-03-28T15:04:16.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.229+0000 7f6330d56640 1 -- 172.21.15.186:0/3317090084 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632402cd50 con 0x7f632c0c7a80 2024-03-28T15:04:16.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.229+0000 7f6330d56640 1 -- 172.21.15.186:0/3317090084 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324034610 con 0x7f632c0c7a80 2024-03-28T15:04:16.230 INFO:tasks.workunit.client.0.smithi186.stdout: create 229/34 file after the snapshot 2024-03-28T15:04:16.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.229+0000 7f6333fbc640 1 -- 172.21.15.186:0/3317090084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c0c7a80 msgr2=0x7f632c06bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.229+0000 7f6333fbc640 1 --2- 172.21.15.186:0/3317090084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c0c7a80 0x7f632c06bc50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f632c070c20 tx=0x7f632402c010 comp rx=0 tx=0).stop 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.229+0000 7f6333fbc640 1 -- 172.21.15.186:0/3317090084 shutdown_connections 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.229+0000 7f6333fbc640 1 --2- 172.21.15.186:0/3317090084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c0c7a80 0x7f632c06bc50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.229+0000 7f6333fbc640 1 --2- 172.21.15.186:0/3317090084 >> v2:172.21.15.191:3300/0 conn(0x7f632c0b1eb0 0x7f632c06b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.229+0000 7f6333fbc640 1 -- 172.21.15.186:0/3317090084 >> 172.21.15.186:0/3317090084 conn(0x7f632c070020 msgr2=0x7f632c070400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.229+0000 7f6333fbc640 1 -- 172.21.15.186:0/3317090084 shutdown_connections 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.229+0000 7f6333fbc640 1 -- 172.21.15.186:0/3317090084 wait complete. 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6333fbc640 1 Processor -- start 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6333fbc640 1 -- start start 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6333fbc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c06c490 0x7f632c1544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6333fbc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c0b1eb0 0x7f632c1e7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:16.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6333fbc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c0c7a80 0x7f632c155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:16.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6333fbc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f632c15b020 con 0x7f632c0c7a80 2024-03-28T15:04:16.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6333fbc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f632c15ad40 con 0x7f632c06c490 2024-03-28T15:04:16.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6333fbc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f632c15b300 con 0x7f632c0b1eb0 2024-03-28T15:04:16.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6331d58640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c06c490 0x7f632c1544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:16.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6332559640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c0c7a80 0x7f632c155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:16.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6331d58640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c06c490 0x7f632c1544e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:40084/0 (socket says 172.21.15.186:40084) 2024-03-28T15:04:16.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6331d58640 1 -- 172.21.15.186:0/198202694 learned_addr learned my addr 172.21.15.186:0/198202694 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:04:16.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.231+0000 7f6331557640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c0b1eb0 0x7f632c1e7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:16.238 INFO:tasks.workunit.client.0.smithi186.stdout: create 229/35 file after the snapshot 2024-03-28T15:04:16.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f6332559640 1 -- 172.21.15.186:0/198202694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c0b1eb0 msgr2=0x7f632c1e7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:16.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f6332559640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c0b1eb0 0x7f632c1e7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:16.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f6332559640 1 -- 172.21.15.186:0/198202694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c06c490 msgr2=0x7f632c1544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:16.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f6332559640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c06c490 0x7f632c1544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:16.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f6332559640 1 -- 172.21.15.186:0/198202694 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632c1f4050 con 0x7f632c0c7a80 2024-03-28T15:04:16.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f6331557640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c0b1eb0 0x7f632c1e7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:04:16.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f6331d58640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c06c490 0x7f632c1544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:04:16.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f6332559640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c0c7a80 0x7f632c155590 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6324032000 tx=0x7f6324004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:16.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f63227fc640 1 -- 172.21.15.186:0/198202694 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324043020 con 0x7f632c0c7a80 2024-03-28T15:04:16.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f63227fc640 1 -- 172.21.15.186:0/198202694 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632403e450 con 0x7f632c0c7a80 2024-03-28T15:04:16.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f6333fbc640 1 -- 172.21.15.186:0/198202694 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f632c1f4330 con 0x7f632c0c7a80 2024-03-28T15:04:16.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f63227fc640 1 -- 172.21.15.186:0/198202694 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324009510 con 0x7f632c0c7a80 2024-03-28T15:04:16.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.232+0000 7f6333fbc640 1 -- 172.21.15.186:0/198202694 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f632c1f48f0 con 0x7f632c0c7a80 2024-03-28T15:04:16.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.233+0000 7f6333fbc640 1 -- 172.21.15.186:0/198202694 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f632c0c8890 con 0x7f632c0c7a80 2024-03-28T15:04:16.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.234+0000 7f63227fc640 1 -- 172.21.15.186:0/198202694 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6324046020 con 0x7f632c0c7a80 2024-03-28T15:04:16.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.234+0000 7f63227fc640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f631006c560 0x7f631006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:04:16.241 INFO:tasks.workunit.client.0.smithi186.stdout: create 229/36 file after the snapshot 2024-03-28T15:04:16.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.234+0000 7f63227fc640 1 -- 172.21.15.186:0/198202694 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5997+0+0 (secure 0 0 0) 0x7f63240adab0 con 0x7f632c0c7a80 2024-03-28T15:04:16.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.234+0000 7f6331d58640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f631006c560 0x7f631006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:04:16.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.235+0000 7f6331d58640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f631006c560 0x7f631006ea20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6328009f50 tx=0x7f6328009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:04:16.242 INFO:tasks.workunit.client.0.smithi186.stdout: create 229/37 file after the snapshot 2024-03-28T15:04:16.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.239+0000 7f63227fc640 1 -- 172.21.15.186:0/198202694 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f632c0c8890 con 0x7f632c0c7a80 2024-03-28T15:04:16.243 INFO:tasks.workunit.client.0.smithi186.stdout: create 229/38 file after the snapshot 2024-03-28T15:04:16.243 INFO:tasks.workunit.client.0.smithi186.stdout: create 229/39 file after the snapshot 2024-03-28T15:04:16.244 INFO:tasks.workunit.client.0.smithi186.stdout: create 230/.snap/snap-subdir-test 2024-03-28T15:04:16.256 INFO:tasks.workunit.client.0.smithi186.stdout: create 230/30 file after the snapshot 2024-03-28T15:04:16.258 INFO:tasks.workunit.client.0.smithi186.stdout: create 230/31 file after the snapshot 2024-03-28T15:04:16.261 INFO:tasks.workunit.client.0.smithi186.stdout: create 230/32 file after the snapshot 2024-03-28T15:04:16.263 INFO:tasks.workunit.client.0.smithi186.stdout: create 230/33 file after the snapshot 2024-03-28T15:04:16.265 INFO:tasks.workunit.client.0.smithi186.stdout: create 230/34 file after the snapshot 2024-03-28T15:04:16.268 INFO:tasks.workunit.client.0.smithi186.stdout: create 230/35 file after the snapshot 2024-03-28T15:04:16.270 INFO:tasks.workunit.client.0.smithi186.stdout: create 230/36 file after the snapshot 2024-03-28T15:04:16.272 INFO:tasks.workunit.client.0.smithi186.stdout: create 230/37 file after the snapshot 2024-03-28T15:04:16.275 INFO:tasks.workunit.client.0.smithi186.stdout: create 230/38 file after the snapshot 2024-03-28T15:04:16.277 INFO:tasks.workunit.client.0.smithi186.stdout: create 230/39 file after the snapshot 2024-03-28T15:04:16.280 INFO:tasks.workunit.client.0.smithi186.stdout: create 231/.snap/snap-subdir-test 2024-03-28T15:04:16.296 INFO:tasks.workunit.client.0.smithi186.stdout: create 231/30 file after the snapshot 2024-03-28T15:04:16.298 INFO:tasks.workunit.client.0.smithi186.stdout: create 231/31 file after the snapshot 2024-03-28T15:04:16.300 INFO:tasks.workunit.client.0.smithi186.stdout: create 231/32 file after the snapshot 2024-03-28T15:04:16.302 INFO:tasks.workunit.client.0.smithi186.stdout: create 231/33 file after the snapshot 2024-03-28T15:04:16.304 INFO:tasks.workunit.client.0.smithi186.stdout: create 231/34 file after the snapshot 2024-03-28T15:04:16.307 INFO:tasks.workunit.client.0.smithi186.stdout: create 231/35 file after the snapshot 2024-03-28T15:04:16.309 INFO:tasks.workunit.client.0.smithi186.stdout: create 231/36 file after the snapshot 2024-03-28T15:04:16.311 INFO:tasks.workunit.client.0.smithi186.stdout: create 231/37 file after the snapshot 2024-03-28T15:04:16.314 INFO:tasks.workunit.client.0.smithi186.stdout: create 231/38 file after the snapshot 2024-03-28T15:04:16.316 INFO:tasks.workunit.client.0.smithi186.stdout: create 231/39 file after the snapshot 2024-03-28T15:04:16.318 INFO:tasks.workunit.client.0.smithi186.stdout: create 232/.snap/snap-subdir-test 2024-03-28T15:04:16.332 INFO:tasks.workunit.client.0.smithi186.stdout: create 232/30 file after the snapshot 2024-03-28T15:04:16.334 INFO:tasks.workunit.client.0.smithi186.stdout: create 232/31 file after the snapshot 2024-03-28T15:04:16.337 INFO:tasks.workunit.client.0.smithi186.stdout: create 232/32 file after the snapshot 2024-03-28T15:04:16.340 INFO:tasks.workunit.client.0.smithi186.stdout: create 232/33 file after the snapshot 2024-03-28T15:04:16.342 INFO:tasks.workunit.client.0.smithi186.stdout: create 232/34 file after the snapshot 2024-03-28T15:04:16.344 INFO:tasks.workunit.client.0.smithi186.stdout: create 232/35 file after the snapshot 2024-03-28T15:04:16.346 INFO:tasks.workunit.client.0.smithi186.stdout: create 232/36 file after the snapshot 2024-03-28T15:04:16.349 INFO:tasks.workunit.client.0.smithi186.stdout: create 232/37 file after the snapshot 2024-03-28T15:04:16.352 INFO:tasks.workunit.client.0.smithi186.stdout: create 232/38 file after the snapshot 2024-03-28T15:04:16.355 INFO:tasks.workunit.client.0.smithi186.stdout: create 232/39 file after the snapshot 2024-03-28T15:04:16.357 INFO:tasks.workunit.client.0.smithi186.stdout: create 233/.snap/snap-subdir-test 2024-03-28T15:04:16.370 INFO:tasks.workunit.client.0.smithi186.stdout: create 233/30 file after the snapshot 2024-03-28T15:04:16.372 INFO:tasks.workunit.client.0.smithi186.stdout: create 233/31 file after the snapshot 2024-03-28T15:04:16.374 INFO:tasks.workunit.client.0.smithi186.stdout: create 233/32 file after the snapshot 2024-03-28T15:04:16.377 INFO:tasks.workunit.client.0.smithi186.stdout: create 233/33 file after the snapshot 2024-03-28T15:04:16.379 INFO:tasks.workunit.client.0.smithi186.stdout: create 233/34 file after the snapshot 2024-03-28T15:04:16.381 INFO:tasks.workunit.client.0.smithi186.stdout: create 233/35 file after the snapshot 2024-03-28T15:04:16.384 INFO:tasks.workunit.client.0.smithi186.stdout: create 233/36 file after the snapshot 2024-03-28T15:04:16.387 INFO:tasks.workunit.client.0.smithi186.stdout: create 233/37 file after the snapshot 2024-03-28T15:04:16.389 INFO:tasks.workunit.client.0.smithi186.stdout: create 233/38 file after the snapshot 2024-03-28T15:04:16.391 INFO:tasks.workunit.client.0.smithi186.stdout: create 233/39 file after the snapshot 2024-03-28T15:04:16.393 INFO:tasks.workunit.client.0.smithi186.stdout: create 234/.snap/snap-subdir-test 2024-03-28T15:04:16.405 INFO:tasks.workunit.client.0.smithi186.stdout: create 234/30 file after the snapshot 2024-03-28T15:04:16.407 INFO:tasks.workunit.client.0.smithi186.stdout: create 234/31 file after the snapshot 2024-03-28T15:04:16.409 INFO:tasks.workunit.client.0.smithi186.stdout: create 234/32 file after the snapshot 2024-03-28T15:04:16.412 INFO:tasks.workunit.client.0.smithi186.stdout: create 234/33 file after the snapshot 2024-03-28T15:04:16.416 INFO:tasks.workunit.client.0.smithi186.stdout: create 234/34 file after the snapshot 2024-03-28T15:04:16.417 INFO:tasks.workunit.client.0.smithi186.stdout: create 234/35 file after the snapshot 2024-03-28T15:04:16.420 INFO:tasks.workunit.client.0.smithi186.stdout: create 234/36 file after the snapshot 2024-03-28T15:04:16.422 INFO:tasks.workunit.client.0.smithi186.stdout: create 234/37 file after the snapshot 2024-03-28T15:04:16.424 INFO:tasks.workunit.client.0.smithi186.stdout: create 234/38 file after the snapshot 2024-03-28T15:04:16.427 INFO:tasks.workunit.client.0.smithi186.stdout: create 234/39 file after the snapshot 2024-03-28T15:04:16.429 INFO:tasks.workunit.client.0.smithi186.stdout: create 235/.snap/snap-subdir-test 2024-03-28T15:04:16.443 INFO:tasks.workunit.client.0.smithi186.stdout: create 235/30 file after the snapshot 2024-03-28T15:04:16.445 INFO:tasks.workunit.client.0.smithi186.stdout: create 235/31 file after the snapshot 2024-03-28T15:04:16.447 INFO:tasks.workunit.client.0.smithi186.stdout: create 235/32 file after the snapshot 2024-03-28T15:04:16.449 INFO:tasks.workunit.client.0.smithi186.stdout: create 235/33 file after the snapshot 2024-03-28T15:04:16.452 INFO:tasks.workunit.client.0.smithi186.stdout: create 235/34 file after the snapshot 2024-03-28T15:04:16.454 INFO:tasks.workunit.client.0.smithi186.stdout: create 235/35 file after the snapshot 2024-03-28T15:04:16.456 INFO:tasks.workunit.client.0.smithi186.stdout: create 235/36 file after the snapshot 2024-03-28T15:04:16.458 INFO:tasks.workunit.client.0.smithi186.stdout: create 235/37 file after the snapshot 2024-03-28T15:04:16.460 INFO:tasks.workunit.client.0.smithi186.stdout: create 235/38 file after the snapshot 2024-03-28T15:04:16.463 INFO:tasks.workunit.client.0.smithi186.stdout: create 235/39 file after the snapshot 2024-03-28T15:04:16.465 INFO:tasks.workunit.client.0.smithi186.stdout: create 236/.snap/snap-subdir-test 2024-03-28T15:04:16.477 INFO:tasks.workunit.client.0.smithi186.stdout: create 236/30 file after the snapshot 2024-03-28T15:04:16.479 INFO:tasks.workunit.client.0.smithi186.stdout: create 236/31 file after the snapshot 2024-03-28T15:04:16.481 INFO:tasks.workunit.client.0.smithi186.stdout: create 236/32 file after the snapshot 2024-03-28T15:04:16.483 INFO:tasks.workunit.client.0.smithi186.stdout: create 236/33 file after the snapshot 2024-03-28T15:04:16.485 INFO:tasks.workunit.client.0.smithi186.stdout: create 236/34 file after the snapshot 2024-03-28T15:04:16.488 INFO:tasks.workunit.client.0.smithi186.stdout: create 236/35 file after the snapshot 2024-03-28T15:04:16.490 INFO:tasks.workunit.client.0.smithi186.stdout: create 236/36 file after the snapshot 2024-03-28T15:04:16.492 INFO:tasks.workunit.client.0.smithi186.stdout: create 236/37 file after the snapshot 2024-03-28T15:04:16.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.495+0000 7f6333fbc640 1 -- 172.21.15.186:0/198202694 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f632c1f4b30 con 0x7f632c0c7a80 2024-03-28T15:04:16.495 INFO:tasks.workunit.client.0.smithi186.stdout: create 236/38 file after the snapshot 2024-03-28T15:04:16.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.495+0000 7f63227fc640 1 -- 172.21.15.186:0/198202694 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6848 (secure 0 0 0) 0x7f632404d070 con 0x7f632c0c7a80 2024-03-28T15:04:16.496 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:04:16.496 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:04:16.496 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 19 2024-03-28T15:04:16.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 -- 172.21.15.186:0/198202694 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f631006c560 msgr2=0x7f631006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:16.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f631006c560 0x7f631006ea20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6328009f50 tx=0x7f6328009190 comp rx=0 tx=0).stop 2024-03-28T15:04:16.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 -- 172.21.15.186:0/198202694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c0c7a80 msgr2=0x7f632c155590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:04:16.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c0c7a80 0x7f632c155590 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6324032000 tx=0x7f6324004a40 comp rx=0 tx=0).stop 2024-03-28T15:04:16.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 -- 172.21.15.186:0/198202694 shutdown_connections 2024-03-28T15:04:16.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f631006c560 0x7f631006ea20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:16.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c0c7a80 0x7f632c155590 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:16.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c0b1eb0 0x7f632c1e7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:16.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 --2- 172.21.15.186:0/198202694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c06c490 0x7f632c1544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:04:16.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 -- 172.21.15.186:0/198202694 >> 172.21.15.186:0/198202694 conn(0x7f632c070020 msgr2=0x7f632c0cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:04:16.499 INFO:tasks.workunit.client.0.smithi186.stdout: create 236/39 file after the snapshot 2024-03-28T15:04:16.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 -- 172.21.15.186:0/198202694 shutdown_connections 2024-03-28T15:04:16.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:04:16.498+0000 7f6333fbc640 1 -- 172.21.15.186:0/198202694 wait complete. 2024-03-28T15:04:16.500 INFO:tasks.workunit.client.0.smithi186.stdout: create 237/.snap/snap-subdir-test 2024-03-28T15:04:16.510 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-03-28T15:04:16.510 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 91 secs before thrashing 2024-03-28T15:04:16.512 INFO:tasks.workunit.client.0.smithi186.stdout: create 237/30 file after the snapshot 2024-03-28T15:04:16.515 INFO:tasks.workunit.client.0.smithi186.stdout: create 237/31 file after the snapshot 2024-03-28T15:04:16.517 INFO:tasks.workunit.client.0.smithi186.stdout: create 237/32 file after the snapshot 2024-03-28T15:04:16.519 INFO:tasks.workunit.client.0.smithi186.stdout: create 237/33 file after the snapshot 2024-03-28T15:04:16.522 INFO:tasks.workunit.client.0.smithi186.stdout: create 237/34 file after the snapshot 2024-03-28T15:04:16.524 INFO:tasks.workunit.client.0.smithi186.stdout: create 237/35 file after the snapshot 2024-03-28T15:04:16.526 INFO:tasks.workunit.client.0.smithi186.stdout: create 237/36 file after the snapshot 2024-03-28T15:04:16.528 INFO:tasks.workunit.client.0.smithi186.stdout: create 237/37 file after the snapshot 2024-03-28T15:04:16.531 INFO:tasks.workunit.client.0.smithi186.stdout: create 237/38 file after the snapshot 2024-03-28T15:04:16.533 INFO:tasks.workunit.client.0.smithi186.stdout: create 237/39 file after the snapshot 2024-03-28T15:04:16.536 INFO:tasks.workunit.client.0.smithi186.stdout: create 238/.snap/snap-subdir-test 2024-03-28T15:04:16.553 INFO:tasks.workunit.client.0.smithi186.stdout: create 238/30 file after the snapshot 2024-03-28T15:04:16.554 INFO:tasks.workunit.client.0.smithi186.stdout: create 238/31 file after the snapshot 2024-03-28T15:04:16.557 INFO:tasks.workunit.client.0.smithi186.stdout: create 238/32 file after the snapshot 2024-03-28T15:04:16.560 INFO:tasks.workunit.client.0.smithi186.stdout: create 238/33 file after the snapshot 2024-03-28T15:04:16.562 INFO:tasks.workunit.client.0.smithi186.stdout: create 238/34 file after the snapshot 2024-03-28T15:04:16.565 INFO:tasks.workunit.client.0.smithi186.stdout: create 238/35 file after the snapshot 2024-03-28T15:04:16.568 INFO:tasks.workunit.client.0.smithi186.stdout: create 238/36 file after the snapshot 2024-03-28T15:04:16.570 INFO:tasks.workunit.client.0.smithi186.stdout: create 238/37 file after the snapshot 2024-03-28T15:04:16.573 INFO:tasks.workunit.client.0.smithi186.stdout: create 238/38 file after the snapshot 2024-03-28T15:04:16.575 INFO:tasks.workunit.client.0.smithi186.stdout: create 238/39 file after the snapshot 2024-03-28T15:04:16.578 INFO:tasks.workunit.client.0.smithi186.stdout: create 239/.snap/snap-subdir-test 2024-03-28T15:04:16.592 INFO:tasks.workunit.client.0.smithi186.stdout: create 239/30 file after the snapshot 2024-03-28T15:04:16.594 INFO:tasks.workunit.client.0.smithi186.stdout: create 239/31 file after the snapshot 2024-03-28T15:04:16.597 INFO:tasks.workunit.client.0.smithi186.stdout: create 239/32 file after the snapshot 2024-03-28T15:04:16.599 INFO:tasks.workunit.client.0.smithi186.stdout: create 239/33 file after the snapshot 2024-03-28T15:04:16.601 INFO:tasks.workunit.client.0.smithi186.stdout: create 239/34 file after the snapshot 2024-03-28T15:04:16.604 INFO:tasks.workunit.client.0.smithi186.stdout: create 239/35 file after the snapshot 2024-03-28T15:04:16.606 INFO:tasks.workunit.client.0.smithi186.stdout: create 239/36 file after the snapshot 2024-03-28T15:04:16.608 INFO:tasks.workunit.client.0.smithi186.stdout: create 239/37 file after the snapshot 2024-03-28T15:04:16.610 INFO:tasks.workunit.client.0.smithi186.stdout: create 239/38 file after the snapshot 2024-03-28T15:04:16.612 INFO:tasks.workunit.client.0.smithi186.stdout: create 239/39 file after the snapshot 2024-03-28T15:04:16.615 INFO:tasks.workunit.client.0.smithi186.stdout: create 240/.snap/snap-subdir-test 2024-03-28T15:04:16.628 INFO:tasks.workunit.client.0.smithi186.stdout: create 240/30 file after the snapshot 2024-03-28T15:04:16.630 INFO:tasks.workunit.client.0.smithi186.stdout: create 240/31 file after the snapshot 2024-03-28T15:04:16.632 INFO:tasks.workunit.client.0.smithi186.stdout: create 240/32 file after the snapshot 2024-03-28T15:04:16.634 INFO:tasks.workunit.client.0.smithi186.stdout: create 240/33 file after the snapshot 2024-03-28T15:04:16.636 INFO:tasks.workunit.client.0.smithi186.stdout: create 240/34 file after the snapshot 2024-03-28T15:04:16.639 INFO:tasks.workunit.client.0.smithi186.stdout: create 240/35 file after the snapshot 2024-03-28T15:04:16.642 INFO:tasks.workunit.client.0.smithi186.stdout: create 240/36 file after the snapshot 2024-03-28T15:04:16.643 INFO:tasks.workunit.client.0.smithi186.stdout: create 240/37 file after the snapshot 2024-03-28T15:04:16.645 INFO:tasks.workunit.client.0.smithi186.stdout: create 240/38 file after the snapshot 2024-03-28T15:04:16.647 INFO:tasks.workunit.client.0.smithi186.stdout: create 240/39 file after the snapshot 2024-03-28T15:04:16.650 INFO:tasks.workunit.client.0.smithi186.stdout: create 241/.snap/snap-subdir-test 2024-03-28T15:04:16.663 INFO:tasks.workunit.client.0.smithi186.stdout: create 241/30 file after the snapshot 2024-03-28T15:04:16.665 INFO:tasks.workunit.client.0.smithi186.stdout: create 241/31 file after the snapshot 2024-03-28T15:04:16.668 INFO:tasks.workunit.client.0.smithi186.stdout: create 241/32 file after the snapshot 2024-03-28T15:04:16.671 INFO:tasks.workunit.client.0.smithi186.stdout: create 241/33 file after the snapshot 2024-03-28T15:04:16.673 INFO:tasks.workunit.client.0.smithi186.stdout: create 241/34 file after the snapshot 2024-03-28T15:04:16.675 INFO:tasks.workunit.client.0.smithi186.stdout: create 241/35 file after the snapshot 2024-03-28T15:04:16.677 INFO:tasks.workunit.client.0.smithi186.stdout: create 241/36 file after the snapshot 2024-03-28T15:04:16.679 INFO:tasks.workunit.client.0.smithi186.stdout: create 241/37 file after the snapshot 2024-03-28T15:04:16.681 INFO:tasks.workunit.client.0.smithi186.stdout: create 241/38 file after the snapshot 2024-03-28T15:04:16.683 INFO:tasks.workunit.client.0.smithi186.stdout: create 241/39 file after the snapshot 2024-03-28T15:04:16.685 INFO:tasks.workunit.client.0.smithi186.stdout: create 242/.snap/snap-subdir-test 2024-03-28T15:04:16.699 INFO:tasks.workunit.client.0.smithi186.stdout: create 242/30 file after the snapshot 2024-03-28T15:04:16.701 INFO:tasks.workunit.client.0.smithi186.stdout: create 242/31 file after the snapshot 2024-03-28T15:04:16.704 INFO:tasks.workunit.client.0.smithi186.stdout: create 242/32 file after the snapshot 2024-03-28T15:04:16.706 INFO:tasks.workunit.client.0.smithi186.stdout: create 242/33 file after the snapshot 2024-03-28T15:04:16.708 INFO:tasks.workunit.client.0.smithi186.stdout: create 242/34 file after the snapshot 2024-03-28T15:04:16.710 INFO:tasks.workunit.client.0.smithi186.stdout: create 242/35 file after the snapshot 2024-03-28T15:04:16.712 INFO:tasks.workunit.client.0.smithi186.stdout: create 242/36 file after the snapshot 2024-03-28T15:04:16.714 INFO:tasks.workunit.client.0.smithi186.stdout: create 242/37 file after the snapshot 2024-03-28T15:04:16.717 INFO:tasks.workunit.client.0.smithi186.stdout: create 242/38 file after the snapshot 2024-03-28T15:04:16.720 INFO:tasks.workunit.client.0.smithi186.stdout: create 242/39 file after the snapshot 2024-03-28T15:04:16.722 INFO:tasks.workunit.client.0.smithi186.stdout: create 243/.snap/snap-subdir-test 2024-03-28T15:04:16.738 INFO:tasks.workunit.client.0.smithi186.stdout: create 243/30 file after the snapshot 2024-03-28T15:04:16.740 INFO:tasks.workunit.client.0.smithi186.stdout: create 243/31 file after the snapshot 2024-03-28T15:04:16.742 INFO:tasks.workunit.client.0.smithi186.stdout: create 243/32 file after the snapshot 2024-03-28T15:04:16.745 INFO:tasks.workunit.client.0.smithi186.stdout: create 243/33 file after the snapshot 2024-03-28T15:04:16.747 INFO:tasks.workunit.client.0.smithi186.stdout: create 243/34 file after the snapshot 2024-03-28T15:04:16.749 INFO:tasks.workunit.client.0.smithi186.stdout: create 243/35 file after the snapshot 2024-03-28T15:04:16.751 INFO:tasks.workunit.client.0.smithi186.stdout: create 243/36 file after the snapshot 2024-03-28T15:04:16.753 INFO:tasks.workunit.client.0.smithi186.stdout: create 243/37 file after the snapshot 2024-03-28T15:04:16.755 INFO:tasks.workunit.client.0.smithi186.stdout: create 243/38 file after the snapshot 2024-03-28T15:04:16.757 INFO:tasks.workunit.client.0.smithi186.stdout: create 243/39 file after the snapshot 2024-03-28T15:04:16.759 INFO:tasks.workunit.client.0.smithi186.stdout: create 244/.snap/snap-subdir-test 2024-03-28T15:04:16.773 INFO:tasks.workunit.client.0.smithi186.stdout: create 244/30 file after the snapshot 2024-03-28T15:04:16.775 INFO:tasks.workunit.client.0.smithi186.stdout: create 244/31 file after the snapshot 2024-03-28T15:04:16.778 INFO:tasks.workunit.client.0.smithi186.stdout: create 244/32 file after the snapshot 2024-03-28T15:04:16.780 INFO:tasks.workunit.client.0.smithi186.stdout: create 244/33 file after the snapshot 2024-03-28T15:04:16.783 INFO:tasks.workunit.client.0.smithi186.stdout: create 244/34 file after the snapshot 2024-03-28T15:04:16.785 INFO:tasks.workunit.client.0.smithi186.stdout: create 244/35 file after the snapshot 2024-03-28T15:04:16.787 INFO:tasks.workunit.client.0.smithi186.stdout: create 244/36 file after the snapshot 2024-03-28T15:04:16.789 INFO:tasks.workunit.client.0.smithi186.stdout: create 244/37 file after the snapshot 2024-03-28T15:04:16.791 INFO:tasks.workunit.client.0.smithi186.stdout: create 244/38 file after the snapshot 2024-03-28T15:04:16.794 INFO:tasks.workunit.client.0.smithi186.stdout: create 244/39 file after the snapshot 2024-03-28T15:04:16.796 INFO:tasks.workunit.client.0.smithi186.stdout: create 245/.snap/snap-subdir-test 2024-03-28T15:04:16.809 INFO:tasks.workunit.client.0.smithi186.stdout: create 245/30 file after the snapshot 2024-03-28T15:04:16.810 INFO:tasks.workunit.client.0.smithi186.stdout: create 245/31 file after the snapshot 2024-03-28T15:04:16.813 INFO:tasks.workunit.client.0.smithi186.stdout: create 245/32 file after the snapshot 2024-03-28T15:04:16.815 INFO:tasks.workunit.client.0.smithi186.stdout: create 245/33 file after the snapshot 2024-03-28T15:04:16.817 INFO:tasks.workunit.client.0.smithi186.stdout: create 245/34 file after the snapshot 2024-03-28T15:04:16.820 INFO:tasks.workunit.client.0.smithi186.stdout: create 245/35 file after the snapshot 2024-03-28T15:04:16.822 INFO:tasks.workunit.client.0.smithi186.stdout: create 245/36 file after the snapshot 2024-03-28T15:04:16.825 INFO:tasks.workunit.client.0.smithi186.stdout: create 245/37 file after the snapshot 2024-03-28T15:04:16.827 INFO:tasks.workunit.client.0.smithi186.stdout: create 245/38 file after the snapshot 2024-03-28T15:04:16.829 INFO:tasks.workunit.client.0.smithi186.stdout: create 245/39 file after the snapshot 2024-03-28T15:04:16.831 INFO:tasks.workunit.client.0.smithi186.stdout: create 246/.snap/snap-subdir-test 2024-03-28T15:04:16.844 INFO:tasks.workunit.client.0.smithi186.stdout: create 246/30 file after the snapshot 2024-03-28T15:04:16.846 INFO:tasks.workunit.client.0.smithi186.stdout: create 246/31 file after the snapshot 2024-03-28T15:04:16.848 INFO:tasks.workunit.client.0.smithi186.stdout: create 246/32 file after the snapshot 2024-03-28T15:04:16.850 INFO:tasks.workunit.client.0.smithi186.stdout: create 246/33 file after the snapshot 2024-03-28T15:04:16.853 INFO:tasks.workunit.client.0.smithi186.stdout: create 246/34 file after the snapshot 2024-03-28T15:04:16.855 INFO:tasks.workunit.client.0.smithi186.stdout: create 246/35 file after the snapshot 2024-03-28T15:04:16.857 INFO:tasks.workunit.client.0.smithi186.stdout: create 246/36 file after the snapshot 2024-03-28T15:04:16.860 INFO:tasks.workunit.client.0.smithi186.stdout: create 246/37 file after the snapshot 2024-03-28T15:04:16.862 INFO:tasks.workunit.client.0.smithi186.stdout: create 246/38 file after the snapshot 2024-03-28T15:04:16.864 INFO:tasks.workunit.client.0.smithi186.stdout: create 246/39 file after the snapshot 2024-03-28T15:04:16.867 INFO:tasks.workunit.client.0.smithi186.stdout: create 247/.snap/snap-subdir-test 2024-03-28T15:04:16.879 INFO:tasks.workunit.client.0.smithi186.stdout: create 247/30 file after the snapshot 2024-03-28T15:04:16.881 INFO:tasks.workunit.client.0.smithi186.stdout: create 247/31 file after the snapshot 2024-03-28T15:04:16.884 INFO:tasks.workunit.client.0.smithi186.stdout: create 247/32 file after the snapshot 2024-03-28T15:04:16.885 INFO:tasks.workunit.client.0.smithi186.stdout: create 247/33 file after the snapshot 2024-03-28T15:04:16.887 INFO:tasks.workunit.client.0.smithi186.stdout: create 247/34 file after the snapshot 2024-03-28T15:04:16.889 INFO:tasks.workunit.client.0.smithi186.stdout: create 247/35 file after the snapshot 2024-03-28T15:04:16.891 INFO:tasks.workunit.client.0.smithi186.stdout: create 247/36 file after the snapshot 2024-03-28T15:04:16.894 INFO:tasks.workunit.client.0.smithi186.stdout: create 247/37 file after the snapshot 2024-03-28T15:04:16.896 INFO:tasks.workunit.client.0.smithi186.stdout: create 247/38 file after the snapshot 2024-03-28T15:04:16.899 INFO:tasks.workunit.client.0.smithi186.stdout: create 247/39 file after the snapshot 2024-03-28T15:04:16.901 INFO:tasks.workunit.client.0.smithi186.stdout: create 248/.snap/snap-subdir-test 2024-03-28T15:04:16.917 INFO:tasks.workunit.client.0.smithi186.stdout: create 248/30 file after the snapshot 2024-03-28T15:04:16.918 INFO:tasks.workunit.client.0.smithi186.stdout: create 248/31 file after the snapshot 2024-03-28T15:04:16.921 INFO:tasks.workunit.client.0.smithi186.stdout: create 248/32 file after the snapshot 2024-03-28T15:04:16.924 INFO:tasks.workunit.client.0.smithi186.stdout: create 248/33 file after the snapshot 2024-03-28T15:04:16.926 INFO:tasks.workunit.client.0.smithi186.stdout: create 248/34 file after the snapshot 2024-03-28T15:04:16.928 INFO:tasks.workunit.client.0.smithi186.stdout: create 248/35 file after the snapshot 2024-03-28T15:04:16.931 INFO:tasks.workunit.client.0.smithi186.stdout: create 248/36 file after the snapshot 2024-03-28T15:04:16.933 INFO:tasks.workunit.client.0.smithi186.stdout: create 248/37 file after the snapshot 2024-03-28T15:04:16.935 INFO:tasks.workunit.client.0.smithi186.stdout: create 248/38 file after the snapshot 2024-03-28T15:04:16.938 INFO:tasks.workunit.client.0.smithi186.stdout: create 248/39 file after the snapshot 2024-03-28T15:04:16.940 INFO:tasks.workunit.client.0.smithi186.stdout: create 249/.snap/snap-subdir-test 2024-03-28T15:04:16.952 INFO:tasks.workunit.client.0.smithi186.stdout: create 249/30 file after the snapshot 2024-03-28T15:04:16.954 INFO:tasks.workunit.client.0.smithi186.stdout: create 249/31 file after the snapshot 2024-03-28T15:04:16.956 INFO:tasks.workunit.client.0.smithi186.stdout: create 249/32 file after the snapshot 2024-03-28T15:04:16.959 INFO:tasks.workunit.client.0.smithi186.stdout: create 249/33 file after the snapshot 2024-03-28T15:04:16.962 INFO:tasks.workunit.client.0.smithi186.stdout: create 249/34 file after the snapshot 2024-03-28T15:04:16.964 INFO:tasks.workunit.client.0.smithi186.stdout: create 249/35 file after the snapshot 2024-03-28T15:04:16.967 INFO:tasks.workunit.client.0.smithi186.stdout: create 249/36 file after the snapshot 2024-03-28T15:04:16.969 INFO:tasks.workunit.client.0.smithi186.stdout: create 249/37 file after the snapshot 2024-03-28T15:04:16.972 INFO:tasks.workunit.client.0.smithi186.stdout: create 249/38 file after the snapshot 2024-03-28T15:04:16.974 INFO:tasks.workunit.client.0.smithi186.stdout: create 249/39 file after the snapshot 2024-03-28T15:04:16.976 INFO:tasks.workunit.client.0.smithi186.stdout: create 250/.snap/snap-subdir-test 2024-03-28T15:04:16.989 INFO:tasks.workunit.client.0.smithi186.stdout: create 250/30 file after the snapshot 2024-03-28T15:04:16.993 INFO:tasks.workunit.client.0.smithi186.stdout: create 250/31 file after the snapshot 2024-03-28T15:04:16.995 INFO:tasks.workunit.client.0.smithi186.stdout: create 250/32 file after the snapshot 2024-03-28T15:04:16.998 INFO:tasks.workunit.client.0.smithi186.stdout: create 250/33 file after the snapshot 2024-03-28T15:04:17.000 INFO:tasks.workunit.client.0.smithi186.stdout: create 250/34 file after the snapshot 2024-03-28T15:04:17.003 INFO:tasks.workunit.client.0.smithi186.stdout: create 250/35 file after the snapshot 2024-03-28T15:04:17.005 INFO:tasks.workunit.client.0.smithi186.stdout: create 250/36 file after the snapshot 2024-03-28T15:04:17.008 INFO:tasks.workunit.client.0.smithi186.stdout: create 250/37 file after the snapshot 2024-03-28T15:04:17.010 INFO:tasks.workunit.client.0.smithi186.stdout: create 250/38 file after the snapshot 2024-03-28T15:04:17.012 INFO:tasks.workunit.client.0.smithi186.stdout: create 250/39 file after the snapshot 2024-03-28T15:04:17.014 INFO:tasks.workunit.client.0.smithi186.stdout: create 251/.snap/snap-subdir-test 2024-03-28T15:04:17.028 INFO:tasks.workunit.client.0.smithi186.stdout: create 251/30 file after the snapshot 2024-03-28T15:04:17.030 INFO:tasks.workunit.client.0.smithi186.stdout: create 251/31 file after the snapshot 2024-03-28T15:04:17.032 INFO:tasks.workunit.client.0.smithi186.stdout: create 251/32 file after the snapshot 2024-03-28T15:04:17.034 INFO:tasks.workunit.client.0.smithi186.stdout: create 251/33 file after the snapshot 2024-03-28T15:04:17.036 INFO:tasks.workunit.client.0.smithi186.stdout: create 251/34 file after the snapshot 2024-03-28T15:04:17.039 INFO:tasks.workunit.client.0.smithi186.stdout: create 251/35 file after the snapshot 2024-03-28T15:04:17.041 INFO:tasks.workunit.client.0.smithi186.stdout: create 251/36 file after the snapshot 2024-03-28T15:04:17.044 INFO:tasks.workunit.client.0.smithi186.stdout: create 251/37 file after the snapshot 2024-03-28T15:04:17.047 INFO:tasks.workunit.client.0.smithi186.stdout: create 251/38 file after the snapshot 2024-03-28T15:04:17.049 INFO:tasks.workunit.client.0.smithi186.stdout: create 251/39 file after the snapshot 2024-03-28T15:04:17.051 INFO:tasks.workunit.client.0.smithi186.stdout: create 252/.snap/snap-subdir-test 2024-03-28T15:04:17.065 INFO:tasks.workunit.client.0.smithi186.stdout: create 252/30 file after the snapshot 2024-03-28T15:04:17.068 INFO:tasks.workunit.client.0.smithi186.stdout: create 252/31 file after the snapshot 2024-03-28T15:04:17.071 INFO:tasks.workunit.client.0.smithi186.stdout: create 252/32 file after the snapshot 2024-03-28T15:04:17.073 INFO:tasks.workunit.client.0.smithi186.stdout: create 252/33 file after the snapshot 2024-03-28T15:04:17.075 INFO:tasks.workunit.client.0.smithi186.stdout: create 252/34 file after the snapshot 2024-03-28T15:04:17.079 INFO:tasks.workunit.client.0.smithi186.stdout: create 252/35 file after the snapshot 2024-03-28T15:04:17.079 INFO:tasks.workunit.client.0.smithi186.stdout: create 252/36 file after the snapshot 2024-03-28T15:04:17.082 INFO:tasks.workunit.client.0.smithi186.stdout: create 252/37 file after the snapshot 2024-03-28T15:04:17.084 INFO:tasks.workunit.client.0.smithi186.stdout: create 252/38 file after the snapshot 2024-03-28T15:04:17.086 INFO:tasks.workunit.client.0.smithi186.stdout: create 252/39 file after the snapshot 2024-03-28T15:04:17.088 INFO:tasks.workunit.client.0.smithi186.stdout: create 253/.snap/snap-subdir-test 2024-03-28T15:04:17.105 INFO:tasks.workunit.client.0.smithi186.stdout: create 253/30 file after the snapshot 2024-03-28T15:04:17.107 INFO:tasks.workunit.client.0.smithi186.stdout: create 253/31 file after the snapshot 2024-03-28T15:04:17.109 INFO:tasks.workunit.client.0.smithi186.stdout: create 253/32 file after the snapshot 2024-03-28T15:04:17.112 INFO:tasks.workunit.client.0.smithi186.stdout: create 253/33 file after the snapshot 2024-03-28T15:04:17.114 INFO:tasks.workunit.client.0.smithi186.stdout: create 253/34 file after the snapshot 2024-03-28T15:04:17.116 INFO:tasks.workunit.client.0.smithi186.stdout: create 253/35 file after the snapshot 2024-03-28T15:04:17.119 INFO:tasks.workunit.client.0.smithi186.stdout: create 253/36 file after the snapshot 2024-03-28T15:04:17.121 INFO:tasks.workunit.client.0.smithi186.stdout: create 253/37 file after the snapshot 2024-03-28T15:04:17.123 INFO:tasks.workunit.client.0.smithi186.stdout: create 253/38 file after the snapshot 2024-03-28T15:04:17.125 INFO:tasks.workunit.client.0.smithi186.stdout: create 253/39 file after the snapshot 2024-03-28T15:04:17.128 INFO:tasks.workunit.client.0.smithi186.stdout: create 254/.snap/snap-subdir-test 2024-03-28T15:04:17.140 INFO:tasks.workunit.client.0.smithi186.stdout: create 254/30 file after the snapshot 2024-03-28T15:04:17.142 INFO:tasks.workunit.client.0.smithi186.stdout: create 254/31 file after the snapshot 2024-03-28T15:04:17.145 INFO:tasks.workunit.client.0.smithi186.stdout: create 254/32 file after the snapshot 2024-03-28T15:04:17.147 INFO:tasks.workunit.client.0.smithi186.stdout: create 254/33 file after the snapshot 2024-03-28T15:04:17.149 INFO:tasks.workunit.client.0.smithi186.stdout: create 254/34 file after the snapshot 2024-03-28T15:04:17.151 INFO:tasks.workunit.client.0.smithi186.stdout: create 254/35 file after the snapshot 2024-03-28T15:04:17.153 INFO:tasks.workunit.client.0.smithi186.stdout: create 254/36 file after the snapshot 2024-03-28T15:04:17.155 INFO:tasks.workunit.client.0.smithi186.stdout: create 254/37 file after the snapshot 2024-03-28T15:04:17.157 INFO:tasks.workunit.client.0.smithi186.stdout: create 254/38 file after the snapshot 2024-03-28T15:04:17.159 INFO:tasks.workunit.client.0.smithi186.stdout: create 254/39 file after the snapshot 2024-03-28T15:04:17.162 INFO:tasks.workunit.client.0.smithi186.stdout: create 255/.snap/snap-subdir-test 2024-03-28T15:04:17.174 INFO:tasks.workunit.client.0.smithi186.stdout: create 255/30 file after the snapshot 2024-03-28T15:04:17.176 INFO:tasks.workunit.client.0.smithi186.stdout: create 255/31 file after the snapshot 2024-03-28T15:04:17.178 INFO:tasks.workunit.client.0.smithi186.stdout: create 255/32 file after the snapshot 2024-03-28T15:04:17.180 INFO:tasks.workunit.client.0.smithi186.stdout: create 255/33 file after the snapshot 2024-03-28T15:04:17.182 INFO:tasks.workunit.client.0.smithi186.stdout: create 255/34 file after the snapshot 2024-03-28T15:04:17.185 INFO:tasks.workunit.client.0.smithi186.stdout: create 255/35 file after the snapshot 2024-03-28T15:04:17.187 INFO:tasks.workunit.client.0.smithi186.stdout: create 255/36 file after the snapshot 2024-03-28T15:04:17.189 INFO:tasks.workunit.client.0.smithi186.stdout: create 255/37 file after the snapshot 2024-03-28T15:04:17.191 INFO:tasks.workunit.client.0.smithi186.stdout: create 255/38 file after the snapshot 2024-03-28T15:04:17.193 INFO:tasks.workunit.client.0.smithi186.stdout: create 255/39 file after the snapshot 2024-03-28T15:04:17.195 INFO:tasks.workunit.client.0.smithi186.stdout: create 256/.snap/snap-subdir-test 2024-03-28T15:04:17.207 INFO:tasks.workunit.client.0.smithi186.stdout: create 256/30 file after the snapshot 2024-03-28T15:04:17.209 INFO:tasks.workunit.client.0.smithi186.stdout: create 256/31 file after the snapshot 2024-03-28T15:04:17.211 INFO:tasks.workunit.client.0.smithi186.stdout: create 256/32 file after the snapshot 2024-03-28T15:04:17.214 INFO:tasks.workunit.client.0.smithi186.stdout: create 256/33 file after the snapshot 2024-03-28T15:04:17.216 INFO:tasks.workunit.client.0.smithi186.stdout: create 256/34 file after the snapshot 2024-03-28T15:04:17.219 INFO:tasks.workunit.client.0.smithi186.stdout: create 256/35 file after the snapshot 2024-03-28T15:04:17.222 INFO:tasks.workunit.client.0.smithi186.stdout: create 256/36 file after the snapshot 2024-03-28T15:04:17.224 INFO:tasks.workunit.client.0.smithi186.stdout: create 256/37 file after the snapshot 2024-03-28T15:04:17.226 INFO:tasks.workunit.client.0.smithi186.stdout: create 256/38 file after the snapshot 2024-03-28T15:04:17.229 INFO:tasks.workunit.client.0.smithi186.stdout: create 256/39 file after the snapshot 2024-03-28T15:04:17.232 INFO:tasks.workunit.client.0.smithi186.stdout: create 257/.snap/snap-subdir-test 2024-03-28T15:04:17.245 INFO:tasks.workunit.client.0.smithi186.stdout: create 257/30 file after the snapshot 2024-03-28T15:04:17.246 INFO:tasks.workunit.client.0.smithi186.stdout: create 257/31 file after the snapshot 2024-03-28T15:04:17.248 INFO:tasks.workunit.client.0.smithi186.stdout: create 257/32 file after the snapshot 2024-03-28T15:04:17.254 INFO:tasks.workunit.client.0.smithi186.stdout: create 257/33 file after the snapshot 2024-03-28T15:04:17.254 INFO:tasks.workunit.client.0.smithi186.stdout: create 257/34 file after the snapshot 2024-03-28T15:04:17.255 INFO:tasks.workunit.client.0.smithi186.stdout: create 257/35 file after the snapshot 2024-03-28T15:04:17.258 INFO:tasks.workunit.client.0.smithi186.stdout: create 257/36 file after the snapshot 2024-03-28T15:04:17.260 INFO:tasks.workunit.client.0.smithi186.stdout: create 257/37 file after the snapshot 2024-03-28T15:04:17.262 INFO:tasks.workunit.client.0.smithi186.stdout: create 257/38 file after the snapshot 2024-03-28T15:04:17.265 INFO:tasks.workunit.client.0.smithi186.stdout: create 257/39 file after the snapshot 2024-03-28T15:04:17.267 INFO:tasks.workunit.client.0.smithi186.stdout: create 258/.snap/snap-subdir-test 2024-03-28T15:04:17.285 INFO:tasks.workunit.client.0.smithi186.stdout: create 258/30 file after the snapshot 2024-03-28T15:04:17.285 INFO:tasks.workunit.client.0.smithi186.stdout: create 258/31 file after the snapshot 2024-03-28T15:04:17.287 INFO:tasks.workunit.client.0.smithi186.stdout: create 258/32 file after the snapshot 2024-03-28T15:04:17.289 INFO:tasks.workunit.client.0.smithi186.stdout: create 258/33 file after the snapshot 2024-03-28T15:04:17.291 INFO:tasks.workunit.client.0.smithi186.stdout: create 258/34 file after the snapshot 2024-03-28T15:04:17.293 INFO:tasks.workunit.client.0.smithi186.stdout: create 258/35 file after the snapshot 2024-03-28T15:04:17.295 INFO:tasks.workunit.client.0.smithi186.stdout: create 258/36 file after the snapshot 2024-03-28T15:04:17.297 INFO:tasks.workunit.client.0.smithi186.stdout: create 258/37 file after the snapshot 2024-03-28T15:04:17.300 INFO:tasks.workunit.client.0.smithi186.stdout: create 258/38 file after the snapshot 2024-03-28T15:04:17.302 INFO:tasks.workunit.client.0.smithi186.stdout: create 258/39 file after the snapshot 2024-03-28T15:04:17.305 INFO:tasks.workunit.client.0.smithi186.stdout: create 259/.snap/snap-subdir-test 2024-03-28T15:04:17.321 INFO:tasks.workunit.client.0.smithi186.stdout: create 259/30 file after the snapshot 2024-03-28T15:04:17.322 INFO:tasks.workunit.client.0.smithi186.stdout: create 259/31 file after the snapshot 2024-03-28T15:04:17.325 INFO:tasks.workunit.client.0.smithi186.stdout: create 259/32 file after the snapshot 2024-03-28T15:04:17.327 INFO:tasks.workunit.client.0.smithi186.stdout: create 259/33 file after the snapshot 2024-03-28T15:04:17.330 INFO:tasks.workunit.client.0.smithi186.stdout: create 259/34 file after the snapshot 2024-03-28T15:04:17.332 INFO:tasks.workunit.client.0.smithi186.stdout: create 259/35 file after the snapshot 2024-03-28T15:04:17.335 INFO:tasks.workunit.client.0.smithi186.stdout: create 259/36 file after the snapshot 2024-03-28T15:04:17.337 INFO:tasks.workunit.client.0.smithi186.stdout: create 259/37 file after the snapshot 2024-03-28T15:04:17.339 INFO:tasks.workunit.client.0.smithi186.stdout: create 259/38 file after the snapshot 2024-03-28T15:04:17.341 INFO:tasks.workunit.client.0.smithi186.stdout: create 259/39 file after the snapshot 2024-03-28T15:04:17.344 INFO:tasks.workunit.client.0.smithi186.stdout: create 260/.snap/snap-subdir-test 2024-03-28T15:04:17.360 INFO:tasks.workunit.client.0.smithi186.stdout: create 260/30 file after the snapshot 2024-03-28T15:04:17.362 INFO:tasks.workunit.client.0.smithi186.stdout: create 260/31 file after the snapshot 2024-03-28T15:04:17.364 INFO:tasks.workunit.client.0.smithi186.stdout: create 260/32 file after the snapshot 2024-03-28T15:04:17.367 INFO:tasks.workunit.client.0.smithi186.stdout: create 260/33 file after the snapshot 2024-03-28T15:04:17.369 INFO:tasks.workunit.client.0.smithi186.stdout: create 260/34 file after the snapshot 2024-03-28T15:04:17.372 INFO:tasks.workunit.client.0.smithi186.stdout: create 260/35 file after the snapshot 2024-03-28T15:04:17.375 INFO:tasks.workunit.client.0.smithi186.stdout: create 260/36 file after the snapshot 2024-03-28T15:04:17.377 INFO:tasks.workunit.client.0.smithi186.stdout: create 260/37 file after the snapshot 2024-03-28T15:04:17.379 INFO:tasks.workunit.client.0.smithi186.stdout: create 260/38 file after the snapshot 2024-03-28T15:04:17.382 INFO:tasks.workunit.client.0.smithi186.stdout: create 260/39 file after the snapshot 2024-03-28T15:04:17.384 INFO:tasks.workunit.client.0.smithi186.stdout: create 261/.snap/snap-subdir-test 2024-03-28T15:04:17.400 INFO:tasks.workunit.client.0.smithi186.stdout: create 261/30 file after the snapshot 2024-03-28T15:04:17.402 INFO:tasks.workunit.client.0.smithi186.stdout: create 261/31 file after the snapshot 2024-03-28T15:04:17.404 INFO:tasks.workunit.client.0.smithi186.stdout: create 261/32 file after the snapshot 2024-03-28T15:04:17.406 INFO:tasks.workunit.client.0.smithi186.stdout: create 261/33 file after the snapshot 2024-03-28T15:04:17.409 INFO:tasks.workunit.client.0.smithi186.stdout: create 261/34 file after the snapshot 2024-03-28T15:04:17.411 INFO:tasks.workunit.client.0.smithi186.stdout: create 261/35 file after the snapshot 2024-03-28T15:04:17.413 INFO:tasks.workunit.client.0.smithi186.stdout: create 261/36 file after the snapshot 2024-03-28T15:04:17.415 INFO:tasks.workunit.client.0.smithi186.stdout: create 261/37 file after the snapshot 2024-03-28T15:04:17.417 INFO:tasks.workunit.client.0.smithi186.stdout: create 261/38 file after the snapshot 2024-03-28T15:04:17.419 INFO:tasks.workunit.client.0.smithi186.stdout: create 261/39 file after the snapshot 2024-03-28T15:04:17.421 INFO:tasks.workunit.client.0.smithi186.stdout: create 262/.snap/snap-subdir-test 2024-03-28T15:04:17.436 INFO:tasks.workunit.client.0.smithi186.stdout: create 262/30 file after the snapshot 2024-03-28T15:04:17.438 INFO:tasks.workunit.client.0.smithi186.stdout: create 262/31 file after the snapshot 2024-03-28T15:04:17.440 INFO:tasks.workunit.client.0.smithi186.stdout: create 262/32 file after the snapshot 2024-03-28T15:04:17.442 INFO:tasks.workunit.client.0.smithi186.stdout: create 262/33 file after the snapshot 2024-03-28T15:04:17.445 INFO:tasks.workunit.client.0.smithi186.stdout: create 262/34 file after the snapshot 2024-03-28T15:04:17.447 INFO:tasks.workunit.client.0.smithi186.stdout: create 262/35 file after the snapshot 2024-03-28T15:04:17.449 INFO:tasks.workunit.client.0.smithi186.stdout: create 262/36 file after the snapshot 2024-03-28T15:04:17.451 INFO:tasks.workunit.client.0.smithi186.stdout: create 262/37 file after the snapshot 2024-03-28T15:04:17.454 INFO:tasks.workunit.client.0.smithi186.stdout: create 262/38 file after the snapshot 2024-03-28T15:04:17.457 INFO:tasks.workunit.client.0.smithi186.stdout: create 262/39 file after the snapshot 2024-03-28T15:04:17.459 INFO:tasks.workunit.client.0.smithi186.stdout: create 263/.snap/snap-subdir-test 2024-03-28T15:04:17.475 INFO:tasks.workunit.client.0.smithi186.stdout: create 263/30 file after the snapshot 2024-03-28T15:04:17.476 INFO:tasks.workunit.client.0.smithi186.stdout: create 263/31 file after the snapshot 2024-03-28T15:04:17.479 INFO:tasks.workunit.client.0.smithi186.stdout: create 263/32 file after the snapshot 2024-03-28T15:04:17.482 INFO:tasks.workunit.client.0.smithi186.stdout: create 263/33 file after the snapshot 2024-03-28T15:04:17.484 INFO:tasks.workunit.client.0.smithi186.stdout: create 263/34 file after the snapshot 2024-03-28T15:04:17.487 INFO:tasks.workunit.client.0.smithi186.stdout: create 263/35 file after the snapshot 2024-03-28T15:04:17.488 INFO:tasks.workunit.client.0.smithi186.stdout: create 263/36 file after the snapshot 2024-03-28T15:04:17.491 INFO:tasks.workunit.client.0.smithi186.stdout: create 263/37 file after the snapshot 2024-03-28T15:04:17.493 INFO:tasks.workunit.client.0.smithi186.stdout: create 263/38 file after the snapshot 2024-03-28T15:04:17.496 INFO:tasks.workunit.client.0.smithi186.stdout: create 263/39 file after the snapshot 2024-03-28T15:04:17.497 INFO:tasks.workunit.client.0.smithi186.stdout: create 264/.snap/snap-subdir-test 2024-03-28T15:04:17.510 INFO:tasks.workunit.client.0.smithi186.stdout: create 264/30 file after the snapshot 2024-03-28T15:04:17.512 INFO:tasks.workunit.client.0.smithi186.stdout: create 264/31 file after the snapshot 2024-03-28T15:04:17.514 INFO:tasks.workunit.client.0.smithi186.stdout: create 264/32 file after the snapshot 2024-03-28T15:04:17.517 INFO:tasks.workunit.client.0.smithi186.stdout: create 264/33 file after the snapshot 2024-03-28T15:04:17.520 INFO:tasks.workunit.client.0.smithi186.stdout: create 264/34 file after the snapshot 2024-03-28T15:04:17.523 INFO:tasks.workunit.client.0.smithi186.stdout: create 264/35 file after the snapshot 2024-03-28T15:04:17.525 INFO:tasks.workunit.client.0.smithi186.stdout: create 264/36 file after the snapshot 2024-03-28T15:04:17.527 INFO:tasks.workunit.client.0.smithi186.stdout: create 264/37 file after the snapshot 2024-03-28T15:04:17.529 INFO:tasks.workunit.client.0.smithi186.stdout: create 264/38 file after the snapshot 2024-03-28T15:04:17.532 INFO:tasks.workunit.client.0.smithi186.stdout: create 264/39 file after the snapshot 2024-03-28T15:04:17.534 INFO:tasks.workunit.client.0.smithi186.stdout: create 265/.snap/snap-subdir-test 2024-03-28T15:04:17.546 INFO:tasks.workunit.client.0.smithi186.stdout: create 265/30 file after the snapshot 2024-03-28T15:04:17.548 INFO:tasks.workunit.client.0.smithi186.stdout: create 265/31 file after the snapshot 2024-03-28T15:04:17.550 INFO:tasks.workunit.client.0.smithi186.stdout: create 265/32 file after the snapshot 2024-03-28T15:04:17.552 INFO:tasks.workunit.client.0.smithi186.stdout: create 265/33 file after the snapshot 2024-03-28T15:04:17.555 INFO:tasks.workunit.client.0.smithi186.stdout: create 265/34 file after the snapshot 2024-03-28T15:04:17.557 INFO:tasks.workunit.client.0.smithi186.stdout: create 265/35 file after the snapshot 2024-03-28T15:04:17.560 INFO:tasks.workunit.client.0.smithi186.stdout: create 265/36 file after the snapshot 2024-03-28T15:04:17.562 INFO:tasks.workunit.client.0.smithi186.stdout: create 265/37 file after the snapshot 2024-03-28T15:04:17.565 INFO:tasks.workunit.client.0.smithi186.stdout: create 265/38 file after the snapshot 2024-03-28T15:04:17.567 INFO:tasks.workunit.client.0.smithi186.stdout: create 265/39 file after the snapshot 2024-03-28T15:04:17.569 INFO:tasks.workunit.client.0.smithi186.stdout: create 266/.snap/snap-subdir-test 2024-03-28T15:04:17.583 INFO:tasks.workunit.client.0.smithi186.stdout: create 266/30 file after the snapshot 2024-03-28T15:04:17.584 INFO:tasks.workunit.client.0.smithi186.stdout: create 266/31 file after the snapshot 2024-03-28T15:04:17.587 INFO:tasks.workunit.client.0.smithi186.stdout: create 266/32 file after the snapshot 2024-03-28T15:04:17.589 INFO:tasks.workunit.client.0.smithi186.stdout: create 266/33 file after the snapshot 2024-03-28T15:04:17.591 INFO:tasks.workunit.client.0.smithi186.stdout: create 266/34 file after the snapshot 2024-03-28T15:04:17.593 INFO:tasks.workunit.client.0.smithi186.stdout: create 266/35 file after the snapshot 2024-03-28T15:04:17.595 INFO:tasks.workunit.client.0.smithi186.stdout: create 266/36 file after the snapshot 2024-03-28T15:04:17.598 INFO:tasks.workunit.client.0.smithi186.stdout: create 266/37 file after the snapshot 2024-03-28T15:04:17.600 INFO:tasks.workunit.client.0.smithi186.stdout: create 266/38 file after the snapshot 2024-03-28T15:04:17.603 INFO:tasks.workunit.client.0.smithi186.stdout: create 266/39 file after the snapshot 2024-03-28T15:04:17.606 INFO:tasks.workunit.client.0.smithi186.stdout: create 267/.snap/snap-subdir-test 2024-03-28T15:04:17.622 INFO:tasks.workunit.client.0.smithi186.stdout: create 267/30 file after the snapshot 2024-03-28T15:04:17.625 INFO:tasks.workunit.client.0.smithi186.stdout: create 267/31 file after the snapshot 2024-03-28T15:04:17.627 INFO:tasks.workunit.client.0.smithi186.stdout: create 267/32 file after the snapshot 2024-03-28T15:04:17.629 INFO:tasks.workunit.client.0.smithi186.stdout: create 267/33 file after the snapshot 2024-03-28T15:04:17.632 INFO:tasks.workunit.client.0.smithi186.stdout: create 267/34 file after the snapshot 2024-03-28T15:04:17.634 INFO:tasks.workunit.client.0.smithi186.stdout: create 267/35 file after the snapshot 2024-03-28T15:04:17.636 INFO:tasks.workunit.client.0.smithi186.stdout: create 267/36 file after the snapshot 2024-03-28T15:04:17.638 INFO:tasks.workunit.client.0.smithi186.stdout: create 267/37 file after the snapshot 2024-03-28T15:04:17.640 INFO:tasks.workunit.client.0.smithi186.stdout: create 267/38 file after the snapshot 2024-03-28T15:04:17.642 INFO:tasks.workunit.client.0.smithi186.stdout: create 267/39 file after the snapshot 2024-03-28T15:04:17.645 INFO:tasks.workunit.client.0.smithi186.stdout: create 268/.snap/snap-subdir-test 2024-03-28T15:04:17.658 INFO:tasks.workunit.client.0.smithi186.stdout: create 268/30 file after the snapshot 2024-03-28T15:04:17.660 INFO:tasks.workunit.client.0.smithi186.stdout: create 268/31 file after the snapshot 2024-03-28T15:04:17.666 INFO:tasks.workunit.client.0.smithi186.stdout: create 268/32 file after the snapshot 2024-03-28T15:04:17.666 INFO:tasks.workunit.client.0.smithi186.stdout: create 268/33 file after the snapshot 2024-03-28T15:04:17.668 INFO:tasks.workunit.client.0.smithi186.stdout: create 268/34 file after the snapshot 2024-03-28T15:04:17.670 INFO:tasks.workunit.client.0.smithi186.stdout: create 268/35 file after the snapshot 2024-03-28T15:04:17.673 INFO:tasks.workunit.client.0.smithi186.stdout: create 268/36 file after the snapshot 2024-03-28T15:04:17.675 INFO:tasks.workunit.client.0.smithi186.stdout: create 268/37 file after the snapshot 2024-03-28T15:04:17.678 INFO:tasks.workunit.client.0.smithi186.stdout: create 268/38 file after the snapshot 2024-03-28T15:04:17.680 INFO:tasks.workunit.client.0.smithi186.stdout: create 268/39 file after the snapshot 2024-03-28T15:04:17.682 INFO:tasks.workunit.client.0.smithi186.stdout: create 269/.snap/snap-subdir-test 2024-03-28T15:04:17.695 INFO:tasks.workunit.client.0.smithi186.stdout: create 269/30 file after the snapshot 2024-03-28T15:04:17.698 INFO:tasks.workunit.client.0.smithi186.stdout: create 269/31 file after the snapshot 2024-03-28T15:04:17.700 INFO:tasks.workunit.client.0.smithi186.stdout: create 269/32 file after the snapshot 2024-03-28T15:04:17.703 INFO:tasks.workunit.client.0.smithi186.stdout: create 269/33 file after the snapshot 2024-03-28T15:04:17.706 INFO:tasks.workunit.client.0.smithi186.stdout: create 269/34 file after the snapshot 2024-03-28T15:04:17.708 INFO:tasks.workunit.client.0.smithi186.stdout: create 269/35 file after the snapshot 2024-03-28T15:04:17.710 INFO:tasks.workunit.client.0.smithi186.stdout: create 269/36 file after the snapshot 2024-03-28T15:04:17.713 INFO:tasks.workunit.client.0.smithi186.stdout: create 269/37 file after the snapshot 2024-03-28T15:04:17.715 INFO:tasks.workunit.client.0.smithi186.stdout: create 269/38 file after the snapshot 2024-03-28T15:04:17.718 INFO:tasks.workunit.client.0.smithi186.stdout: create 269/39 file after the snapshot 2024-03-28T15:04:17.720 INFO:tasks.workunit.client.0.smithi186.stdout: create 270/.snap/snap-subdir-test 2024-03-28T15:04:17.736 INFO:tasks.workunit.client.0.smithi186.stdout: create 270/30 file after the snapshot 2024-03-28T15:04:17.738 INFO:tasks.workunit.client.0.smithi186.stdout: create 270/31 file after the snapshot 2024-03-28T15:04:17.740 INFO:tasks.workunit.client.0.smithi186.stdout: create 270/32 file after the snapshot 2024-03-28T15:04:17.744 INFO:tasks.workunit.client.0.smithi186.stdout: create 270/33 file after the snapshot 2024-03-28T15:04:17.746 INFO:tasks.workunit.client.0.smithi186.stdout: create 270/34 file after the snapshot 2024-03-28T15:04:17.748 INFO:tasks.workunit.client.0.smithi186.stdout: create 270/35 file after the snapshot 2024-03-28T15:04:17.750 INFO:tasks.workunit.client.0.smithi186.stdout: create 270/36 file after the snapshot 2024-03-28T15:04:17.752 INFO:tasks.workunit.client.0.smithi186.stdout: create 270/37 file after the snapshot 2024-03-28T15:04:17.754 INFO:tasks.workunit.client.0.smithi186.stdout: create 270/38 file after the snapshot 2024-03-28T15:04:17.757 INFO:tasks.workunit.client.0.smithi186.stdout: create 270/39 file after the snapshot 2024-03-28T15:04:17.759 INFO:tasks.workunit.client.0.smithi186.stdout: create 271/.snap/snap-subdir-test 2024-03-28T15:04:17.773 INFO:tasks.workunit.client.0.smithi186.stdout: create 271/30 file after the snapshot 2024-03-28T15:04:17.776 INFO:tasks.workunit.client.0.smithi186.stdout: create 271/31 file after the snapshot 2024-03-28T15:04:17.778 INFO:tasks.workunit.client.0.smithi186.stdout: create 271/32 file after the snapshot 2024-03-28T15:04:17.780 INFO:tasks.workunit.client.0.smithi186.stdout: create 271/33 file after the snapshot 2024-03-28T15:04:17.782 INFO:tasks.workunit.client.0.smithi186.stdout: create 271/34 file after the snapshot 2024-03-28T15:04:17.784 INFO:tasks.workunit.client.0.smithi186.stdout: create 271/35 file after the snapshot 2024-03-28T15:04:17.786 INFO:tasks.workunit.client.0.smithi186.stdout: create 271/36 file after the snapshot 2024-03-28T15:04:17.788 INFO:tasks.workunit.client.0.smithi186.stdout: create 271/37 file after the snapshot 2024-03-28T15:04:17.790 INFO:tasks.workunit.client.0.smithi186.stdout: create 271/38 file after the snapshot 2024-03-28T15:04:17.793 INFO:tasks.workunit.client.0.smithi186.stdout: create 271/39 file after the snapshot 2024-03-28T15:04:17.795 INFO:tasks.workunit.client.0.smithi186.stdout: create 272/.snap/snap-subdir-test 2024-03-28T15:04:17.811 INFO:tasks.workunit.client.0.smithi186.stdout: create 272/30 file after the snapshot 2024-03-28T15:04:17.814 INFO:tasks.workunit.client.0.smithi186.stdout: create 272/31 file after the snapshot 2024-03-28T15:04:17.816 INFO:tasks.workunit.client.0.smithi186.stdout: create 272/32 file after the snapshot 2024-03-28T15:04:17.818 INFO:tasks.workunit.client.0.smithi186.stdout: create 272/33 file after the snapshot 2024-03-28T15:04:17.820 INFO:tasks.workunit.client.0.smithi186.stdout: create 272/34 file after the snapshot 2024-03-28T15:04:17.823 INFO:tasks.workunit.client.0.smithi186.stdout: create 272/35 file after the snapshot 2024-03-28T15:04:17.825 INFO:tasks.workunit.client.0.smithi186.stdout: create 272/36 file after the snapshot 2024-03-28T15:04:17.828 INFO:tasks.workunit.client.0.smithi186.stdout: create 272/37 file after the snapshot 2024-03-28T15:04:17.830 INFO:tasks.workunit.client.0.smithi186.stdout: create 272/38 file after the snapshot 2024-03-28T15:04:17.832 INFO:tasks.workunit.client.0.smithi186.stdout: create 272/39 file after the snapshot 2024-03-28T15:04:17.834 INFO:tasks.workunit.client.0.smithi186.stdout: create 273/.snap/snap-subdir-test 2024-03-28T15:04:17.850 INFO:tasks.workunit.client.0.smithi186.stdout: create 273/30 file after the snapshot 2024-03-28T15:04:17.853 INFO:tasks.workunit.client.0.smithi186.stdout: create 273/31 file after the snapshot 2024-03-28T15:04:17.855 INFO:tasks.workunit.client.0.smithi186.stdout: create 273/32 file after the snapshot 2024-03-28T15:04:17.858 INFO:tasks.workunit.client.0.smithi186.stdout: create 273/33 file after the snapshot 2024-03-28T15:04:17.861 INFO:tasks.workunit.client.0.smithi186.stdout: create 273/34 file after the snapshot 2024-03-28T15:04:17.863 INFO:tasks.workunit.client.0.smithi186.stdout: create 273/35 file after the snapshot 2024-03-28T15:04:17.866 INFO:tasks.workunit.client.0.smithi186.stdout: create 273/36 file after the snapshot 2024-03-28T15:04:17.870 INFO:tasks.workunit.client.0.smithi186.stdout: create 273/37 file after the snapshot 2024-03-28T15:04:17.870 INFO:tasks.workunit.client.0.smithi186.stdout: create 273/38 file after the snapshot 2024-03-28T15:04:17.873 INFO:tasks.workunit.client.0.smithi186.stdout: create 273/39 file after the snapshot 2024-03-28T15:04:17.876 INFO:tasks.workunit.client.0.smithi186.stdout: create 274/.snap/snap-subdir-test 2024-03-28T15:04:17.893 INFO:tasks.workunit.client.0.smithi186.stdout: create 274/30 file after the snapshot 2024-03-28T15:04:17.895 INFO:tasks.workunit.client.0.smithi186.stdout: create 274/31 file after the snapshot 2024-03-28T15:04:17.898 INFO:tasks.workunit.client.0.smithi186.stdout: create 274/32 file after the snapshot 2024-03-28T15:04:17.900 INFO:tasks.workunit.client.0.smithi186.stdout: create 274/33 file after the snapshot 2024-03-28T15:04:17.903 INFO:tasks.workunit.client.0.smithi186.stdout: create 274/34 file after the snapshot 2024-03-28T15:04:17.906 INFO:tasks.workunit.client.0.smithi186.stdout: create 274/35 file after the snapshot 2024-03-28T15:04:17.909 INFO:tasks.workunit.client.0.smithi186.stdout: create 274/36 file after the snapshot 2024-03-28T15:04:17.911 INFO:tasks.workunit.client.0.smithi186.stdout: create 274/37 file after the snapshot 2024-03-28T15:04:17.913 INFO:tasks.workunit.client.0.smithi186.stdout: create 274/38 file after the snapshot 2024-03-28T15:04:17.916 INFO:tasks.workunit.client.0.smithi186.stdout: create 274/39 file after the snapshot 2024-03-28T15:04:17.918 INFO:tasks.workunit.client.0.smithi186.stdout: create 275/.snap/snap-subdir-test 2024-03-28T15:04:17.931 INFO:tasks.workunit.client.0.smithi186.stdout: create 275/30 file after the snapshot 2024-03-28T15:04:17.933 INFO:tasks.workunit.client.0.smithi186.stdout: create 275/31 file after the snapshot 2024-03-28T15:04:17.936 INFO:tasks.workunit.client.0.smithi186.stdout: create 275/32 file after the snapshot 2024-03-28T15:04:17.939 INFO:tasks.workunit.client.0.smithi186.stdout: create 275/33 file after the snapshot 2024-03-28T15:04:17.941 INFO:tasks.workunit.client.0.smithi186.stdout: create 275/34 file after the snapshot 2024-03-28T15:04:17.943 INFO:tasks.workunit.client.0.smithi186.stdout: create 275/35 file after the snapshot 2024-03-28T15:04:17.945 INFO:tasks.workunit.client.0.smithi186.stdout: create 275/36 file after the snapshot 2024-03-28T15:04:17.948 INFO:tasks.workunit.client.0.smithi186.stdout: create 275/37 file after the snapshot 2024-03-28T15:04:17.951 INFO:tasks.workunit.client.0.smithi186.stdout: create 275/38 file after the snapshot 2024-03-28T15:04:17.953 INFO:tasks.workunit.client.0.smithi186.stdout: create 275/39 file after the snapshot 2024-03-28T15:04:17.955 INFO:tasks.workunit.client.0.smithi186.stdout: create 276/.snap/snap-subdir-test 2024-03-28T15:04:17.968 INFO:tasks.workunit.client.0.smithi186.stdout: create 276/30 file after the snapshot 2024-03-28T15:04:17.970 INFO:tasks.workunit.client.0.smithi186.stdout: create 276/31 file after the snapshot 2024-03-28T15:04:17.972 INFO:tasks.workunit.client.0.smithi186.stdout: create 276/32 file after the snapshot 2024-03-28T15:04:17.974 INFO:tasks.workunit.client.0.smithi186.stdout: create 276/33 file after the snapshot 2024-03-28T15:04:17.976 INFO:tasks.workunit.client.0.smithi186.stdout: create 276/34 file after the snapshot 2024-03-28T15:04:17.979 INFO:tasks.workunit.client.0.smithi186.stdout: create 276/35 file after the snapshot 2024-03-28T15:04:17.981 INFO:tasks.workunit.client.0.smithi186.stdout: create 276/36 file after the snapshot 2024-03-28T15:04:17.983 INFO:tasks.workunit.client.0.smithi186.stdout: create 276/37 file after the snapshot 2024-03-28T15:04:17.985 INFO:tasks.workunit.client.0.smithi186.stdout: create 276/38 file after the snapshot 2024-03-28T15:04:17.988 INFO:tasks.workunit.client.0.smithi186.stdout: create 276/39 file after the snapshot 2024-03-28T15:04:17.990 INFO:tasks.workunit.client.0.smithi186.stdout: create 277/.snap/snap-subdir-test 2024-03-28T15:04:18.007 INFO:tasks.workunit.client.0.smithi186.stdout: create 277/30 file after the snapshot 2024-03-28T15:04:18.010 INFO:tasks.workunit.client.0.smithi186.stdout: create 277/31 file after the snapshot 2024-03-28T15:04:18.012 INFO:tasks.workunit.client.0.smithi186.stdout: create 277/32 file after the snapshot 2024-03-28T15:04:18.016 INFO:tasks.workunit.client.0.smithi186.stdout: create 277/33 file after the snapshot 2024-03-28T15:04:18.016 INFO:tasks.workunit.client.0.smithi186.stdout: create 277/34 file after the snapshot 2024-03-28T15:04:18.018 INFO:tasks.workunit.client.0.smithi186.stdout: create 277/35 file after the snapshot 2024-03-28T15:04:18.023 INFO:tasks.workunit.client.0.smithi186.stdout: create 277/36 file after the snapshot 2024-03-28T15:04:18.024 INFO:tasks.workunit.client.0.smithi186.stdout: create 277/37 file after the snapshot 2024-03-28T15:04:18.027 INFO:tasks.workunit.client.0.smithi186.stdout: create 277/38 file after the snapshot 2024-03-28T15:04:18.029 INFO:tasks.workunit.client.0.smithi186.stdout: create 277/39 file after the snapshot 2024-03-28T15:04:18.031 INFO:tasks.workunit.client.0.smithi186.stdout: create 278/.snap/snap-subdir-test 2024-03-28T15:04:18.044 INFO:tasks.workunit.client.0.smithi186.stdout: create 278/30 file after the snapshot 2024-03-28T15:04:18.046 INFO:tasks.workunit.client.0.smithi186.stdout: create 278/31 file after the snapshot 2024-03-28T15:04:18.048 INFO:tasks.workunit.client.0.smithi186.stdout: create 278/32 file after the snapshot 2024-03-28T15:04:18.050 INFO:tasks.workunit.client.0.smithi186.stdout: create 278/33 file after the snapshot 2024-03-28T15:04:18.052 INFO:tasks.workunit.client.0.smithi186.stdout: create 278/34 file after the snapshot 2024-03-28T15:04:18.054 INFO:tasks.workunit.client.0.smithi186.stdout: create 278/35 file after the snapshot 2024-03-28T15:04:18.057 INFO:tasks.workunit.client.0.smithi186.stdout: create 278/36 file after the snapshot 2024-03-28T15:04:18.060 INFO:tasks.workunit.client.0.smithi186.stdout: create 278/37 file after the snapshot 2024-03-28T15:04:18.061 INFO:tasks.workunit.client.0.smithi186.stdout: create 278/38 file after the snapshot 2024-03-28T15:04:18.064 INFO:tasks.workunit.client.0.smithi186.stdout: create 278/39 file after the snapshot 2024-03-28T15:04:18.066 INFO:tasks.workunit.client.0.smithi186.stdout: create 279/.snap/snap-subdir-test 2024-03-28T15:04:18.079 INFO:tasks.workunit.client.0.smithi186.stdout: create 279/30 file after the snapshot 2024-03-28T15:04:18.082 INFO:tasks.workunit.client.0.smithi186.stdout: create 279/31 file after the snapshot 2024-03-28T15:04:18.085 INFO:tasks.workunit.client.0.smithi186.stdout: create 279/32 file after the snapshot 2024-03-28T15:04:18.087 INFO:tasks.workunit.client.0.smithi186.stdout: create 279/33 file after the snapshot 2024-03-28T15:04:18.089 INFO:tasks.workunit.client.0.smithi186.stdout: create 279/34 file after the snapshot 2024-03-28T15:04:18.092 INFO:tasks.workunit.client.0.smithi186.stdout: create 279/35 file after the snapshot 2024-03-28T15:04:18.095 INFO:tasks.workunit.client.0.smithi186.stdout: create 279/36 file after the snapshot 2024-03-28T15:04:18.097 INFO:tasks.workunit.client.0.smithi186.stdout: create 279/37 file after the snapshot 2024-03-28T15:04:18.099 INFO:tasks.workunit.client.0.smithi186.stdout: create 279/38 file after the snapshot 2024-03-28T15:04:18.101 INFO:tasks.workunit.client.0.smithi186.stdout: create 279/39 file after the snapshot 2024-03-28T15:04:18.104 INFO:tasks.workunit.client.0.smithi186.stdout: create 280/.snap/snap-subdir-test 2024-03-28T15:04:18.118 INFO:tasks.workunit.client.0.smithi186.stdout: create 280/30 file after the snapshot 2024-03-28T15:04:18.120 INFO:tasks.workunit.client.0.smithi186.stdout: create 280/31 file after the snapshot 2024-03-28T15:04:18.123 INFO:tasks.workunit.client.0.smithi186.stdout: create 280/32 file after the snapshot 2024-03-28T15:04:18.125 INFO:tasks.workunit.client.0.smithi186.stdout: create 280/33 file after the snapshot 2024-03-28T15:04:18.128 INFO:tasks.workunit.client.0.smithi186.stdout: create 280/34 file after the snapshot 2024-03-28T15:04:18.130 INFO:tasks.workunit.client.0.smithi186.stdout: create 280/35 file after the snapshot 2024-03-28T15:04:18.132 INFO:tasks.workunit.client.0.smithi186.stdout: create 280/36 file after the snapshot 2024-03-28T15:04:18.134 INFO:tasks.workunit.client.0.smithi186.stdout: create 280/37 file after the snapshot 2024-03-28T15:04:18.137 INFO:tasks.workunit.client.0.smithi186.stdout: create 280/38 file after the snapshot 2024-03-28T15:04:18.139 INFO:tasks.workunit.client.0.smithi186.stdout: create 280/39 file after the snapshot 2024-03-28T15:04:18.141 INFO:tasks.workunit.client.0.smithi186.stdout: create 281/.snap/snap-subdir-test 2024-03-28T15:04:18.154 INFO:tasks.workunit.client.0.smithi186.stdout: create 281/30 file after the snapshot 2024-03-28T15:04:18.156 INFO:tasks.workunit.client.0.smithi186.stdout: create 281/31 file after the snapshot 2024-03-28T15:04:18.158 INFO:tasks.workunit.client.0.smithi186.stdout: create 281/32 file after the snapshot 2024-03-28T15:04:18.160 INFO:tasks.workunit.client.0.smithi186.stdout: create 281/33 file after the snapshot 2024-03-28T15:04:18.162 INFO:tasks.workunit.client.0.smithi186.stdout: create 281/34 file after the snapshot 2024-03-28T15:04:18.164 INFO:tasks.workunit.client.0.smithi186.stdout: create 281/35 file after the snapshot 2024-03-28T15:04:18.167 INFO:tasks.workunit.client.0.smithi186.stdout: create 281/36 file after the snapshot 2024-03-28T15:04:18.169 INFO:tasks.workunit.client.0.smithi186.stdout: create 281/37 file after the snapshot 2024-03-28T15:04:18.171 INFO:tasks.workunit.client.0.smithi186.stdout: create 281/38 file after the snapshot 2024-03-28T15:04:18.173 INFO:tasks.workunit.client.0.smithi186.stdout: create 281/39 file after the snapshot 2024-03-28T15:04:18.176 INFO:tasks.workunit.client.0.smithi186.stdout: create 282/.snap/snap-subdir-test 2024-03-28T15:04:18.189 INFO:tasks.workunit.client.0.smithi186.stdout: create 282/30 file after the snapshot 2024-03-28T15:04:18.191 INFO:tasks.workunit.client.0.smithi186.stdout: create 282/31 file after the snapshot 2024-03-28T15:04:18.193 INFO:tasks.workunit.client.0.smithi186.stdout: create 282/32 file after the snapshot 2024-03-28T15:04:18.196 INFO:tasks.workunit.client.0.smithi186.stdout: create 282/33 file after the snapshot 2024-03-28T15:04:18.198 INFO:tasks.workunit.client.0.smithi186.stdout: create 282/34 file after the snapshot 2024-03-28T15:04:18.200 INFO:tasks.workunit.client.0.smithi186.stdout: create 282/35 file after the snapshot 2024-03-28T15:04:18.202 INFO:tasks.workunit.client.0.smithi186.stdout: create 282/36 file after the snapshot 2024-03-28T15:04:18.204 INFO:tasks.workunit.client.0.smithi186.stdout: create 282/37 file after the snapshot 2024-03-28T15:04:18.207 INFO:tasks.workunit.client.0.smithi186.stdout: create 282/38 file after the snapshot 2024-03-28T15:04:18.210 INFO:tasks.workunit.client.0.smithi186.stdout: create 282/39 file after the snapshot 2024-03-28T15:04:18.212 INFO:tasks.workunit.client.0.smithi186.stdout: create 283/.snap/snap-subdir-test 2024-03-28T15:04:18.226 INFO:tasks.workunit.client.0.smithi186.stdout: create 283/30 file after the snapshot 2024-03-28T15:04:18.229 INFO:tasks.workunit.client.0.smithi186.stdout: create 283/31 file after the snapshot 2024-03-28T15:04:18.232 INFO:tasks.workunit.client.0.smithi186.stdout: create 283/32 file after the snapshot 2024-03-28T15:04:18.234 INFO:tasks.workunit.client.0.smithi186.stdout: create 283/33 file after the snapshot 2024-03-28T15:04:18.236 INFO:tasks.workunit.client.0.smithi186.stdout: create 283/34 file after the snapshot 2024-03-28T15:04:18.238 INFO:tasks.workunit.client.0.smithi186.stdout: create 283/35 file after the snapshot 2024-03-28T15:04:18.241 INFO:tasks.workunit.client.0.smithi186.stdout: create 283/36 file after the snapshot 2024-03-28T15:04:18.243 INFO:tasks.workunit.client.0.smithi186.stdout: create 283/37 file after the snapshot 2024-03-28T15:04:18.246 INFO:tasks.workunit.client.0.smithi186.stdout: create 283/38 file after the snapshot 2024-03-28T15:04:18.248 INFO:tasks.workunit.client.0.smithi186.stdout: create 283/39 file after the snapshot 2024-03-28T15:04:18.251 INFO:tasks.workunit.client.0.smithi186.stdout: create 284/.snap/snap-subdir-test 2024-03-28T15:04:18.267 INFO:tasks.workunit.client.0.smithi186.stdout: create 284/30 file after the snapshot 2024-03-28T15:04:18.269 INFO:tasks.workunit.client.0.smithi186.stdout: create 284/31 file after the snapshot 2024-03-28T15:04:18.272 INFO:tasks.workunit.client.0.smithi186.stdout: create 284/32 file after the snapshot 2024-03-28T15:04:18.278 INFO:tasks.workunit.client.0.smithi186.stdout: create 284/33 file after the snapshot 2024-03-28T15:04:18.278 INFO:tasks.workunit.client.0.smithi186.stdout: create 284/34 file after the snapshot 2024-03-28T15:04:18.280 INFO:tasks.workunit.client.0.smithi186.stdout: create 284/35 file after the snapshot 2024-03-28T15:04:18.282 INFO:tasks.workunit.client.0.smithi186.stdout: create 284/36 file after the snapshot 2024-03-28T15:04:18.284 INFO:tasks.workunit.client.0.smithi186.stdout: create 284/37 file after the snapshot 2024-03-28T15:04:18.287 INFO:tasks.workunit.client.0.smithi186.stdout: create 284/38 file after the snapshot 2024-03-28T15:04:18.289 INFO:tasks.workunit.client.0.smithi186.stdout: create 284/39 file after the snapshot 2024-03-28T15:04:18.291 INFO:tasks.workunit.client.0.smithi186.stdout: create 285/.snap/snap-subdir-test 2024-03-28T15:04:18.292 INFO:tasks.ceph.mds.f.smithi191.stderr:2024-03-28T15:04:18.292+0000 7f9b3526f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T15:04:18.304 INFO:tasks.workunit.client.0.smithi186.stdout: create 285/30 file after the snapshot 2024-03-28T15:04:18.306 INFO:tasks.workunit.client.0.smithi186.stdout: create 285/31 file after the snapshot 2024-03-28T15:04:18.309 INFO:tasks.workunit.client.0.smithi186.stdout: create 285/32 file after the snapshot 2024-03-28T15:04:18.311 INFO:tasks.workunit.client.0.smithi186.stdout: create 285/33 file after the snapshot 2024-03-28T15:04:18.314 INFO:tasks.workunit.client.0.smithi186.stdout: create 285/34 file after the snapshot 2024-03-28T15:04:18.316 INFO:tasks.workunit.client.0.smithi186.stdout: create 285/35 file after the snapshot 2024-03-28T15:04:18.318 INFO:tasks.workunit.client.0.smithi186.stdout: create 285/36 file after the snapshot 2024-03-28T15:04:18.320 INFO:tasks.workunit.client.0.smithi186.stdout: create 285/37 file after the snapshot 2024-03-28T15:04:18.322 INFO:tasks.workunit.client.0.smithi186.stdout: create 285/38 file after the snapshot 2024-03-28T15:04:18.324 INFO:tasks.workunit.client.0.smithi186.stdout: create 285/39 file after the snapshot 2024-03-28T15:04:18.327 INFO:tasks.workunit.client.0.smithi186.stdout: create 286/.snap/snap-subdir-test 2024-03-28T15:04:18.340 INFO:tasks.workunit.client.0.smithi186.stdout: create 286/30 file after the snapshot 2024-03-28T15:04:18.342 INFO:tasks.workunit.client.0.smithi186.stdout: create 286/31 file after the snapshot 2024-03-28T15:04:18.344 INFO:tasks.workunit.client.0.smithi186.stdout: create 286/32 file after the snapshot 2024-03-28T15:04:18.347 INFO:tasks.workunit.client.0.smithi186.stdout: create 286/33 file after the snapshot 2024-03-28T15:04:18.349 INFO:tasks.workunit.client.0.smithi186.stdout: create 286/34 file after the snapshot 2024-03-28T15:04:18.352 INFO:tasks.workunit.client.0.smithi186.stdout: create 286/35 file after the snapshot 2024-03-28T15:04:18.354 INFO:tasks.workunit.client.0.smithi186.stdout: create 286/36 file after the snapshot 2024-03-28T15:04:18.357 INFO:tasks.workunit.client.0.smithi186.stdout: create 286/37 file after the snapshot 2024-03-28T15:04:18.359 INFO:tasks.workunit.client.0.smithi186.stdout: create 286/38 file after the snapshot 2024-03-28T15:04:18.361 INFO:tasks.workunit.client.0.smithi186.stdout: create 286/39 file after the snapshot 2024-03-28T15:04:18.363 INFO:tasks.workunit.client.0.smithi186.stdout: create 287/.snap/snap-subdir-test 2024-03-28T15:04:18.376 INFO:tasks.workunit.client.0.smithi186.stdout: create 287/30 file after the snapshot 2024-03-28T15:04:18.378 INFO:tasks.workunit.client.0.smithi186.stdout: create 287/31 file after the snapshot 2024-03-28T15:04:18.380 INFO:tasks.workunit.client.0.smithi186.stdout: create 287/32 file after the snapshot 2024-03-28T15:04:18.383 INFO:tasks.workunit.client.0.smithi186.stdout: create 287/33 file after the snapshot 2024-03-28T15:04:18.385 INFO:tasks.workunit.client.0.smithi186.stdout: create 287/34 file after the snapshot 2024-03-28T15:04:18.388 INFO:tasks.workunit.client.0.smithi186.stdout: create 287/35 file after the snapshot 2024-03-28T15:04:18.390 INFO:tasks.workunit.client.0.smithi186.stdout: create 287/36 file after the snapshot 2024-03-28T15:04:18.392 INFO:tasks.workunit.client.0.smithi186.stdout: create 287/37 file after the snapshot 2024-03-28T15:04:18.394 INFO:tasks.workunit.client.0.smithi186.stdout: create 287/38 file after the snapshot 2024-03-28T15:04:18.396 INFO:tasks.workunit.client.0.smithi186.stdout: create 287/39 file after the snapshot 2024-03-28T15:04:18.399 INFO:tasks.workunit.client.0.smithi186.stdout: create 288/.snap/snap-subdir-test 2024-03-28T15:04:18.413 INFO:tasks.workunit.client.0.smithi186.stdout: create 288/30 file after the snapshot 2024-03-28T15:04:18.415 INFO:tasks.workunit.client.0.smithi186.stdout: create 288/31 file after the snapshot 2024-03-28T15:04:18.418 INFO:tasks.workunit.client.0.smithi186.stdout: create 288/32 file after the snapshot 2024-03-28T15:04:18.421 INFO:tasks.workunit.client.0.smithi186.stdout: create 288/33 file after the snapshot 2024-03-28T15:04:18.423 INFO:tasks.workunit.client.0.smithi186.stdout: create 288/34 file after the snapshot 2024-03-28T15:04:18.426 INFO:tasks.workunit.client.0.smithi186.stdout: create 288/35 file after the snapshot 2024-03-28T15:04:18.428 INFO:tasks.workunit.client.0.smithi186.stdout: create 288/36 file after the snapshot 2024-03-28T15:04:18.430 INFO:tasks.workunit.client.0.smithi186.stdout: create 288/37 file after the snapshot 2024-03-28T15:04:18.432 INFO:tasks.workunit.client.0.smithi186.stdout: create 288/38 file after the snapshot 2024-03-28T15:04:18.435 INFO:tasks.workunit.client.0.smithi186.stdout: create 288/39 file after the snapshot 2024-03-28T15:04:18.438 INFO:tasks.workunit.client.0.smithi186.stdout: create 289/.snap/snap-subdir-test 2024-03-28T15:04:18.453 INFO:tasks.workunit.client.0.smithi186.stdout: create 289/30 file after the snapshot 2024-03-28T15:04:18.454 INFO:tasks.workunit.client.0.smithi186.stdout: create 289/31 file after the snapshot 2024-03-28T15:04:18.457 INFO:tasks.workunit.client.0.smithi186.stdout: create 289/32 file after the snapshot 2024-03-28T15:04:18.459 INFO:tasks.workunit.client.0.smithi186.stdout: create 289/33 file after the snapshot 2024-03-28T15:04:18.461 INFO:tasks.workunit.client.0.smithi186.stdout: create 289/34 file after the snapshot 2024-03-28T15:04:18.464 INFO:tasks.workunit.client.0.smithi186.stdout: create 289/35 file after the snapshot 2024-03-28T15:04:18.467 INFO:tasks.workunit.client.0.smithi186.stdout: create 289/36 file after the snapshot 2024-03-28T15:04:18.470 INFO:tasks.workunit.client.0.smithi186.stdout: create 289/37 file after the snapshot 2024-03-28T15:04:18.471 INFO:tasks.workunit.client.0.smithi186.stdout: create 289/38 file after the snapshot 2024-03-28T15:04:18.473 INFO:tasks.workunit.client.0.smithi186.stdout: create 289/39 file after the snapshot 2024-03-28T15:04:18.475 INFO:tasks.workunit.client.0.smithi186.stdout: create 290/.snap/snap-subdir-test 2024-03-28T15:04:18.488 INFO:tasks.workunit.client.0.smithi186.stdout: create 290/30 file after the snapshot 2024-03-28T15:04:18.490 INFO:tasks.workunit.client.0.smithi186.stdout: create 290/31 file after the snapshot 2024-03-28T15:04:18.492 INFO:tasks.workunit.client.0.smithi186.stdout: create 290/32 file after the snapshot 2024-03-28T15:04:18.495 INFO:tasks.workunit.client.0.smithi186.stdout: create 290/33 file after the snapshot 2024-03-28T15:04:18.498 INFO:tasks.workunit.client.0.smithi186.stdout: create 290/34 file after the snapshot 2024-03-28T15:04:18.500 INFO:tasks.workunit.client.0.smithi186.stdout: create 290/35 file after the snapshot 2024-03-28T15:04:18.502 INFO:tasks.workunit.client.0.smithi186.stdout: create 290/36 file after the snapshot 2024-03-28T15:04:18.505 INFO:tasks.workunit.client.0.smithi186.stdout: create 290/37 file after the snapshot 2024-03-28T15:04:18.507 INFO:tasks.workunit.client.0.smithi186.stdout: create 290/38 file after the snapshot 2024-03-28T15:04:18.509 INFO:tasks.workunit.client.0.smithi186.stdout: create 290/39 file after the snapshot 2024-03-28T15:04:18.512 INFO:tasks.workunit.client.0.smithi186.stdout: create 291/.snap/snap-subdir-test 2024-03-28T15:04:18.526 INFO:tasks.workunit.client.0.smithi186.stdout: create 291/30 file after the snapshot 2024-03-28T15:04:18.529 INFO:tasks.workunit.client.0.smithi186.stdout: create 291/31 file after the snapshot 2024-03-28T15:04:18.532 INFO:tasks.workunit.client.0.smithi186.stdout: create 291/32 file after the snapshot 2024-03-28T15:04:18.534 INFO:tasks.workunit.client.0.smithi186.stdout: create 291/33 file after the snapshot 2024-03-28T15:04:18.536 INFO:tasks.workunit.client.0.smithi186.stdout: create 291/34 file after the snapshot 2024-03-28T15:04:18.539 INFO:tasks.workunit.client.0.smithi186.stdout: create 291/35 file after the snapshot 2024-03-28T15:04:18.541 INFO:tasks.workunit.client.0.smithi186.stdout: create 291/36 file after the snapshot 2024-03-28T15:04:18.543 INFO:tasks.workunit.client.0.smithi186.stdout: create 291/37 file after the snapshot 2024-03-28T15:04:18.545 INFO:tasks.workunit.client.0.smithi186.stdout: create 291/38 file after the snapshot 2024-03-28T15:04:18.547 INFO:tasks.workunit.client.0.smithi186.stdout: create 291/39 file after the snapshot 2024-03-28T15:04:18.550 INFO:tasks.workunit.client.0.smithi186.stdout: create 292/.snap/snap-subdir-test 2024-03-28T15:04:18.563 INFO:tasks.workunit.client.0.smithi186.stdout: create 292/30 file after the snapshot 2024-03-28T15:04:18.565 INFO:tasks.workunit.client.0.smithi186.stdout: create 292/31 file after the snapshot 2024-03-28T15:04:18.568 INFO:tasks.workunit.client.0.smithi186.stdout: create 292/32 file after the snapshot 2024-03-28T15:04:18.571 INFO:tasks.workunit.client.0.smithi186.stdout: create 292/33 file after the snapshot 2024-03-28T15:04:18.573 INFO:tasks.workunit.client.0.smithi186.stdout: create 292/34 file after the snapshot 2024-03-28T15:04:18.575 INFO:tasks.workunit.client.0.smithi186.stdout: create 292/35 file after the snapshot 2024-03-28T15:04:18.577 INFO:tasks.workunit.client.0.smithi186.stdout: create 292/36 file after the snapshot 2024-03-28T15:04:18.580 INFO:tasks.workunit.client.0.smithi186.stdout: create 292/37 file after the snapshot 2024-03-28T15:04:18.582 INFO:tasks.workunit.client.0.smithi186.stdout: create 292/38 file after the snapshot 2024-03-28T15:04:18.585 INFO:tasks.workunit.client.0.smithi186.stdout: create 292/39 file after the snapshot 2024-03-28T15:04:18.587 INFO:tasks.workunit.client.0.smithi186.stdout: create 293/.snap/snap-subdir-test 2024-03-28T15:04:18.603 INFO:tasks.workunit.client.0.smithi186.stdout: create 293/30 file after the snapshot 2024-03-28T15:04:18.605 INFO:tasks.workunit.client.0.smithi186.stdout: create 293/31 file after the snapshot 2024-03-28T15:04:18.607 INFO:tasks.workunit.client.0.smithi186.stdout: create 293/32 file after the snapshot 2024-03-28T15:04:18.609 INFO:tasks.workunit.client.0.smithi186.stdout: create 293/33 file after the snapshot 2024-03-28T15:04:18.612 INFO:tasks.workunit.client.0.smithi186.stdout: create 293/34 file after the snapshot 2024-03-28T15:04:18.615 INFO:tasks.workunit.client.0.smithi186.stdout: create 293/35 file after the snapshot 2024-03-28T15:04:18.617 INFO:tasks.workunit.client.0.smithi186.stdout: create 293/36 file after the snapshot 2024-03-28T15:04:18.619 INFO:tasks.workunit.client.0.smithi186.stdout: create 293/37 file after the snapshot 2024-03-28T15:04:18.622 INFO:tasks.workunit.client.0.smithi186.stdout: create 293/38 file after the snapshot 2024-03-28T15:04:18.625 INFO:tasks.workunit.client.0.smithi186.stdout: create 293/39 file after the snapshot 2024-03-28T15:04:18.627 INFO:tasks.workunit.client.0.smithi186.stdout: create 294/.snap/snap-subdir-test 2024-03-28T15:04:18.641 INFO:tasks.workunit.client.0.smithi186.stdout: create 294/30 file after the snapshot 2024-03-28T15:04:18.643 INFO:tasks.workunit.client.0.smithi186.stdout: create 294/31 file after the snapshot 2024-03-28T15:04:18.645 INFO:tasks.workunit.client.0.smithi186.stdout: create 294/32 file after the snapshot 2024-03-28T15:04:18.648 INFO:tasks.workunit.client.0.smithi186.stdout: create 294/33 file after the snapshot 2024-03-28T15:04:18.650 INFO:tasks.workunit.client.0.smithi186.stdout: create 294/34 file after the snapshot 2024-03-28T15:04:18.652 INFO:tasks.workunit.client.0.smithi186.stdout: create 294/35 file after the snapshot 2024-03-28T15:04:18.654 INFO:tasks.workunit.client.0.smithi186.stdout: create 294/36 file after the snapshot 2024-03-28T15:04:18.657 INFO:tasks.workunit.client.0.smithi186.stdout: create 294/37 file after the snapshot 2024-03-28T15:04:18.660 INFO:tasks.workunit.client.0.smithi186.stdout: create 294/38 file after the snapshot 2024-03-28T15:04:18.663 INFO:tasks.workunit.client.0.smithi186.stdout: create 294/39 file after the snapshot 2024-03-28T15:04:18.665 INFO:tasks.workunit.client.0.smithi186.stdout: create 295/.snap/snap-subdir-test 2024-03-28T15:04:18.682 INFO:tasks.workunit.client.0.smithi186.stdout: create 295/30 file after the snapshot 2024-03-28T15:04:18.684 INFO:tasks.workunit.client.0.smithi186.stdout: create 295/31 file after the snapshot 2024-03-28T15:04:18.686 INFO:tasks.workunit.client.0.smithi186.stdout: create 295/32 file after the snapshot 2024-03-28T15:04:18.688 INFO:tasks.workunit.client.0.smithi186.stdout: create 295/33 file after the snapshot 2024-03-28T15:04:18.694 INFO:tasks.workunit.client.0.smithi186.stdout: create 295/34 file after the snapshot 2024-03-28T15:04:18.694 INFO:tasks.workunit.client.0.smithi186.stdout: create 295/35 file after the snapshot 2024-03-28T15:04:18.696 INFO:tasks.workunit.client.0.smithi186.stdout: create 295/36 file after the snapshot 2024-03-28T15:04:18.699 INFO:tasks.workunit.client.0.smithi186.stdout: create 295/37 file after the snapshot 2024-03-28T15:04:18.700 INFO:tasks.workunit.client.0.smithi186.stdout: create 295/38 file after the snapshot 2024-03-28T15:04:18.702 INFO:tasks.workunit.client.0.smithi186.stdout: create 295/39 file after the snapshot 2024-03-28T15:04:18.704 INFO:tasks.workunit.client.0.smithi186.stdout: create 296/.snap/snap-subdir-test 2024-03-28T15:04:18.717 INFO:tasks.workunit.client.0.smithi186.stdout: create 296/30 file after the snapshot 2024-03-28T15:04:18.719 INFO:tasks.workunit.client.0.smithi186.stdout: create 296/31 file after the snapshot 2024-03-28T15:04:18.722 INFO:tasks.workunit.client.0.smithi186.stdout: create 296/32 file after the snapshot 2024-03-28T15:04:18.724 INFO:tasks.workunit.client.0.smithi186.stdout: create 296/33 file after the snapshot 2024-03-28T15:04:18.727 INFO:tasks.workunit.client.0.smithi186.stdout: create 296/34 file after the snapshot 2024-03-28T15:04:18.729 INFO:tasks.workunit.client.0.smithi186.stdout: create 296/35 file after the snapshot 2024-03-28T15:04:18.731 INFO:tasks.workunit.client.0.smithi186.stdout: create 296/36 file after the snapshot 2024-03-28T15:04:18.734 INFO:tasks.workunit.client.0.smithi186.stdout: create 296/37 file after the snapshot 2024-03-28T15:04:18.736 INFO:tasks.workunit.client.0.smithi186.stdout: create 296/38 file after the snapshot 2024-03-28T15:04:18.738 INFO:tasks.workunit.client.0.smithi186.stdout: create 296/39 file after the snapshot 2024-03-28T15:04:18.740 INFO:tasks.workunit.client.0.smithi186.stdout: create 297/.snap/snap-subdir-test 2024-03-28T15:04:18.756 INFO:tasks.workunit.client.0.smithi186.stdout: create 297/30 file after the snapshot 2024-03-28T15:04:18.758 INFO:tasks.workunit.client.0.smithi186.stdout: create 297/31 file after the snapshot 2024-03-28T15:04:18.760 INFO:tasks.workunit.client.0.smithi186.stdout: create 297/32 file after the snapshot 2024-03-28T15:04:18.762 INFO:tasks.workunit.client.0.smithi186.stdout: create 297/33 file after the snapshot 2024-03-28T15:04:18.764 INFO:tasks.workunit.client.0.smithi186.stdout: create 297/34 file after the snapshot 2024-03-28T15:04:18.767 INFO:tasks.workunit.client.0.smithi186.stdout: create 297/35 file after the snapshot 2024-03-28T15:04:18.769 INFO:tasks.workunit.client.0.smithi186.stdout: create 297/36 file after the snapshot 2024-03-28T15:04:18.771 INFO:tasks.workunit.client.0.smithi186.stdout: create 297/37 file after the snapshot 2024-03-28T15:04:18.773 INFO:tasks.workunit.client.0.smithi186.stdout: create 297/38 file after the snapshot 2024-03-28T15:04:18.776 INFO:tasks.workunit.client.0.smithi186.stdout: create 297/39 file after the snapshot 2024-03-28T15:04:18.778 INFO:tasks.workunit.client.0.smithi186.stdout: create 298/.snap/snap-subdir-test 2024-03-28T15:04:18.791 INFO:tasks.workunit.client.0.smithi186.stdout: create 298/30 file after the snapshot 2024-03-28T15:04:18.794 INFO:tasks.workunit.client.0.smithi186.stdout: create 298/31 file after the snapshot 2024-03-28T15:04:18.796 INFO:tasks.workunit.client.0.smithi186.stdout: create 298/32 file after the snapshot 2024-03-28T15:04:18.798 INFO:tasks.workunit.client.0.smithi186.stdout: create 298/33 file after the snapshot 2024-03-28T15:04:18.801 INFO:tasks.workunit.client.0.smithi186.stdout: create 298/34 file after the snapshot 2024-03-28T15:04:18.803 INFO:tasks.workunit.client.0.smithi186.stdout: create 298/35 file after the snapshot 2024-03-28T15:04:18.805 INFO:tasks.workunit.client.0.smithi186.stdout: create 298/36 file after the snapshot 2024-03-28T15:04:18.807 INFO:tasks.workunit.client.0.smithi186.stdout: create 298/37 file after the snapshot 2024-03-28T15:04:18.809 INFO:tasks.workunit.client.0.smithi186.stdout: create 298/38 file after the snapshot 2024-03-28T15:04:18.811 INFO:tasks.workunit.client.0.smithi186.stdout: create 298/39 file after the snapshot 2024-03-28T15:04:18.813 INFO:tasks.workunit.client.0.smithi186.stdout: create 299/.snap/snap-subdir-test 2024-03-28T15:04:18.826 INFO:tasks.workunit.client.0.smithi186.stdout: create 299/30 file after the snapshot 2024-03-28T15:04:18.828 INFO:tasks.workunit.client.0.smithi186.stdout: create 299/31 file after the snapshot 2024-03-28T15:04:18.831 INFO:tasks.workunit.client.0.smithi186.stdout: create 299/32 file after the snapshot 2024-03-28T15:04:18.833 INFO:tasks.workunit.client.0.smithi186.stdout: create 299/33 file after the snapshot 2024-03-28T15:04:18.835 INFO:tasks.workunit.client.0.smithi186.stdout: create 299/34 file after the snapshot 2024-03-28T15:04:18.837 INFO:tasks.workunit.client.0.smithi186.stdout: create 299/35 file after the snapshot 2024-03-28T15:04:18.839 INFO:tasks.workunit.client.0.smithi186.stdout: create 299/36 file after the snapshot 2024-03-28T15:04:18.842 INFO:tasks.workunit.client.0.smithi186.stdout: create 299/37 file after the snapshot 2024-03-28T15:04:18.844 INFO:tasks.workunit.client.0.smithi186.stdout: create 299/38 file after the snapshot 2024-03-28T15:04:18.846 INFO:tasks.workunit.client.0.smithi186.stdout: create 299/39 file after the snapshot 2024-03-28T15:04:18.849 INFO:tasks.workunit.client.0.smithi186.stdout:Take second snapshot .snap/test2 2024-03-28T15:04:18.884 INFO:tasks.workunit.client.0.smithi186.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2024-03-28T15:04:19.654 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2024-03-28T15:04:19.655 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2024-03-28T15:04:19.655 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2024-03-28T15:04:19.655 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2024-03-28T15:04:19.655 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2024-03-28T15:04:19.655 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2024-03-28T15:04:19.655 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2024-03-28T15:04:19.655 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2024-03-28T15:04:19.655 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2024-03-28T15:04:19.655 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2024-03-28T15:04:19.655 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2024-03-28T15:04:19.656 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2024-03-28T15:04:19.657 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2024-03-28T15:04:19.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2024-03-28T15:04:19.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2024-03-28T15:04:19.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2024-03-28T15:04:19.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2024-03-28T15:04:19.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2024-03-28T15:04:19.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2024-03-28T15:04:19.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2024-03-28T15:04:19.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2024-03-28T15:04:19.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2024-03-28T15:04:19.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2024-03-28T15:04:19.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2024-03-28T15:04:20.094 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2024-03-28T15:04:20.095 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2024-03-28T15:04:20.096 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2024-03-28T15:04:20.097 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2024-03-28T15:04:20.098 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2024-03-28T15:04:20.099 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2024-03-28T15:04:20.100 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2024-03-28T15:04:20.100 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2024-03-28T15:04:20.100 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2024-03-28T15:04:20.100 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2024-03-28T15:04:20.100 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2024-03-28T15:04:20.100 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2024-03-28T15:04:20.100 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2024-03-28T15:04:20.100 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2024-03-28T15:04:20.100 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2024-03-28T15:04:20.567 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2024-03-28T15:04:20.568 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2024-03-28T15:04:20.569 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2024-03-28T15:04:20.570 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2024-03-28T15:04:20.571 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2024-03-28T15:04:20.572 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2024-03-28T15:04:20.573 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2024-03-28T15:04:20.573 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2024-03-28T15:04:20.573 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2024-03-28T15:04:20.573 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2024-03-28T15:04:20.573 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2024-03-28T15:04:20.573 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2024-03-28T15:04:20.573 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2024-03-28T15:04:20.573 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2024-03-28T15:04:20.573 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2024-03-28T15:04:21.042 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2024-03-28T15:04:21.043 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2024-03-28T15:04:21.043 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2024-03-28T15:04:21.043 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2024-03-28T15:04:21.043 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2024-03-28T15:04:21.043 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2024-03-28T15:04:21.043 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2024-03-28T15:04:21.043 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2024-03-28T15:04:21.043 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2024-03-28T15:04:21.043 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2024-03-28T15:04:21.043 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2024-03-28T15:04:21.044 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2024-03-28T15:04:21.044 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2024-03-28T15:04:21.044 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2024-03-28T15:04:21.044 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2024-03-28T15:04:21.044 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2024-03-28T15:04:21.044 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2024-03-28T15:04:21.044 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2024-03-28T15:04:21.044 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2024-03-28T15:04:21.044 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2024-03-28T15:04:21.044 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2024-03-28T15:04:21.044 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2024-03-28T15:04:21.045 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2024-03-28T15:04:21.045 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2024-03-28T15:04:21.045 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2024-03-28T15:04:21.045 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2024-03-28T15:04:21.045 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2024-03-28T15:04:21.045 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2024-03-28T15:04:21.045 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2024-03-28T15:04:21.045 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2024-03-28T15:04:21.045 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2024-03-28T15:04:21.045 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2024-03-28T15:04:21.045 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2024-03-28T15:04:21.046 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2024-03-28T15:04:21.046 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2024-03-28T15:04:21.046 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2024-03-28T15:04:21.046 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2024-03-28T15:04:21.046 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2024-03-28T15:04:21.046 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2024-03-28T15:04:21.046 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2024-03-28T15:04:21.046 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2024-03-28T15:04:21.046 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2024-03-28T15:04:21.046 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2024-03-28T15:04:21.046 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2024-03-28T15:04:21.047 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2024-03-28T15:04:21.048 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2024-03-28T15:04:21.048 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2024-03-28T15:04:21.048 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2024-03-28T15:04:21.048 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2024-03-28T15:04:21.048 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2024-03-28T15:04:21.048 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2024-03-28T15:04:21.048 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2024-03-28T15:04:21.048 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2024-03-28T15:04:21.048 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2024-03-28T15:04:21.048 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2024-03-28T15:04:21.048 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2024-03-28T15:04:21.049 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2024-03-28T15:04:21.050 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2024-03-28T15:04:21.050 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2024-03-28T15:04:21.494 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2024-03-28T15:04:21.495 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2024-03-28T15:04:21.496 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2024-03-28T15:04:21.497 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2024-03-28T15:04:21.498 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2024-03-28T15:04:21.499 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2024-03-28T15:04:21.499 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2024-03-28T15:04:21.499 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2024-03-28T15:04:21.499 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2024-03-28T15:04:21.499 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2024-03-28T15:04:21.499 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2024-03-28T15:04:21.499 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2024-03-28T15:04:21.499 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2024-03-28T15:04:21.499 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2024-03-28T15:04:21.499 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2024-03-28T15:04:21.499 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2024-03-28T15:04:21.947 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2024-03-28T15:04:21.948 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2024-03-28T15:04:21.949 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2024-03-28T15:04:21.950 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2024-03-28T15:04:21.951 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2024-03-28T15:04:21.952 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2024-03-28T15:04:21.953 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2024-03-28T15:04:21.953 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2024-03-28T15:04:21.953 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2024-03-28T15:04:21.953 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2024-03-28T15:04:21.953 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2024-03-28T15:04:21.953 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2024-03-28T15:04:21.953 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2024-03-28T15:04:21.953 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2024-03-28T15:04:21.953 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2024-03-28T15:04:21.953 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2024-03-28T15:04:22.500 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2024-03-28T15:04:22.500 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2024-03-28T15:04:22.500 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2024-03-28T15:04:22.500 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2024-03-28T15:04:22.500 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2024-03-28T15:04:22.500 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2024-03-28T15:04:22.500 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2024-03-28T15:04:22.500 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2024-03-28T15:04:22.500 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2024-03-28T15:04:22.501 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2024-03-28T15:04:22.502 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2024-03-28T15:04:22.503 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2024-03-28T15:04:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2024-03-28T15:04:22.505 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2024-03-28T15:04:22.958 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2024-03-28T15:04:22.958 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2024-03-28T15:04:22.958 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2024-03-28T15:04:22.958 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2024-03-28T15:04:22.958 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2024-03-28T15:04:22.958 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2024-03-28T15:04:22.958 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2024-03-28T15:04:22.958 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2024-03-28T15:04:22.959 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2024-03-28T15:04:22.960 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2024-03-28T15:04:22.961 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2024-03-28T15:04:22.962 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2024-03-28T15:04:22.963 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2024-03-28T15:04:23.585 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2024-03-28T15:04:23.585 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2024-03-28T15:04:23.585 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2024-03-28T15:04:23.585 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2024-03-28T15:04:23.585 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2024-03-28T15:04:23.586 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2024-03-28T15:04:23.587 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2024-03-28T15:04:23.588 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2024-03-28T15:04:23.589 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2024-03-28T15:04:23.590 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2024-03-28T15:04:23.591 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2024-03-28T15:04:24.132 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2024-03-28T15:04:24.132 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2024-03-28T15:04:24.132 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2024-03-28T15:04:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2024-03-28T15:04:24.134 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2024-03-28T15:04:24.135 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2024-03-28T15:04:24.136 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2024-03-28T15:04:24.137 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2024-03-28T15:04:24.137 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2024-03-28T15:04:24.137 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2024-03-28T15:04:24.137 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2024-03-28T15:04:24.137 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2024-03-28T15:04:24.137 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2024-03-28T15:04:24.137 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2024-03-28T15:04:24.137 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2024-03-28T15:04:24.137 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2024-03-28T15:04:24.137 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2024-03-28T15:04:24.137 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2024-03-28T15:04:24.138 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2024-03-28T15:04:24.138 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2024-03-28T15:04:24.138 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2024-03-28T15:04:24.138 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2024-03-28T15:04:24.138 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2024-03-28T15:04:24.138 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2024-03-28T15:04:24.138 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2024-03-28T15:04:24.593 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2024-03-28T15:04:24.593 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2024-03-28T15:04:24.593 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2024-03-28T15:04:24.593 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2024-03-28T15:04:24.593 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2024-03-28T15:04:24.593 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2024-03-28T15:04:24.594 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2024-03-28T15:04:24.595 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2024-03-28T15:04:24.596 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2024-03-28T15:04:24.597 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2024-03-28T15:04:24.598 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2024-03-28T15:04:24.599 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2024-03-28T15:04:24.599 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2024-03-28T15:04:24.599 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2024-03-28T15:04:24.599 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2024-03-28T15:04:25.139 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2024-03-28T15:04:25.139 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2024-03-28T15:04:25.139 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2024-03-28T15:04:25.139 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2024-03-28T15:04:25.139 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2024-03-28T15:04:25.139 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2024-03-28T15:04:25.139 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2024-03-28T15:04:25.140 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2024-03-28T15:04:25.141 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2024-03-28T15:04:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2024-03-28T15:04:25.143 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2024-03-28T15:04:25.144 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2024-03-28T15:04:25.144 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2024-03-28T15:04:25.144 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2024-03-28T15:04:25.144 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2024-03-28T15:04:25.144 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2024-03-28T15:04:25.144 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2024-03-28T15:04:25.144 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2024-03-28T15:04:25.144 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2024-03-28T15:04:25.144 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2024-03-28T15:04:25.144 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2024-03-28T15:04:25.144 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2024-03-28T15:04:25.145 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2024-03-28T15:04:25.145 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2024-03-28T15:04:25.730 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:04:25.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:04:25.770 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2024-03-28T15:04:25.770 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2024-03-28T15:04:25.770 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2024-03-28T15:04:25.770 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2024-03-28T15:04:25.770 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2024-03-28T15:04:25.770 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2024-03-28T15:04:25.770 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2024-03-28T15:04:25.770 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2024-03-28T15:04:25.770 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2024-03-28T15:04:25.770 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2024-03-28T15:04:25.771 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2024-03-28T15:04:25.772 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2024-03-28T15:04:25.773 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2024-03-28T15:04:25.774 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2024-03-28T15:04:25.775 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2024-03-28T15:04:26.292 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2024-03-28T15:04:26.292 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2024-03-28T15:04:26.292 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2024-03-28T15:04:26.292 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2024-03-28T15:04:26.292 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2024-03-28T15:04:26.292 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2024-03-28T15:04:26.292 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2024-03-28T15:04:26.292 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2024-03-28T15:04:26.293 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2024-03-28T15:04:26.294 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2024-03-28T15:04:26.295 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2024-03-28T15:04:26.296 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2024-03-28T15:04:26.297 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2024-03-28T15:04:26.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2024-03-28T15:04:26.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2024-03-28T15:04:26.658 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2024-03-28T15:04:26.659 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2024-03-28T15:04:26.660 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2024-03-28T15:04:26.661 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2024-03-28T15:04:26.662 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2024-03-28T15:04:26.663 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2024-03-28T15:04:26.663 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2024-03-28T15:04:26.663 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2024-03-28T15:04:26.663 INFO:tasks.workunit.client.0.smithi186.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2024-03-28T15:04:26.663 INFO:tasks.workunit.client.0.smithi186.stdout:Take third snapshot .snap/test3 2024-03-28T15:04:26.778 INFO:tasks.workunit.client.0.smithi186.stdout:Delete the snapshots... 2024-03-28T15:04:27.942 INFO:tasks.workunit.client.0.smithi186.stdout:.//.snap/test1 2024-03-28T15:04:27.942 INFO:tasks.workunit.client.0.smithi186.stdout:.//.snap/test2 2024-03-28T15:04:27.942 INFO:tasks.workunit.client.0.smithi186.stdout:.//.snap/test3 2024-03-28T15:04:27.949 INFO:tasks.workunit.client.0.smithi186.stdout:./274/.snap/snap-subdir-test 2024-03-28T15:04:27.958 INFO:tasks.workunit.client.0.smithi186.stdout:./166/.snap/snap-subdir-test 2024-03-28T15:04:27.966 INFO:tasks.workunit.client.0.smithi186.stdout:./256/.snap/snap-subdir-test 2024-03-28T15:04:27.975 INFO:tasks.workunit.client.0.smithi186.stdout:./247/.snap/snap-subdir-test 2024-03-28T15:04:27.983 INFO:tasks.workunit.client.0.smithi186.stdout:./248/.snap/snap-subdir-test 2024-03-28T15:04:27.992 INFO:tasks.workunit.client.0.smithi186.stdout:./147/.snap/snap-subdir-test 2024-03-28T15:04:28.000 INFO:tasks.workunit.client.0.smithi186.stdout:./151/.snap/snap-subdir-test 2024-03-28T15:04:28.009 INFO:tasks.workunit.client.0.smithi186.stdout:./170/.snap/snap-subdir-test 2024-03-28T15:04:28.017 INFO:tasks.workunit.client.0.smithi186.stdout:./167/.snap/snap-subdir-test 2024-03-28T15:04:28.026 INFO:tasks.workunit.client.0.smithi186.stdout:./140/.snap/snap-subdir-test 2024-03-28T15:04:28.034 INFO:tasks.workunit.client.0.smithi186.stdout:./142/.snap/snap-subdir-test 2024-03-28T15:04:28.042 INFO:tasks.workunit.client.0.smithi186.stdout:./150/.snap/snap-subdir-test 2024-03-28T15:04:28.051 INFO:tasks.workunit.client.0.smithi186.stdout:./273/.snap/snap-subdir-test 2024-03-28T15:04:28.059 INFO:tasks.workunit.client.0.smithi186.stdout:./269/.snap/snap-subdir-test 2024-03-28T15:04:28.068 INFO:tasks.workunit.client.0.smithi186.stdout:./105/.snap/snap-subdir-test 2024-03-28T15:04:28.129 INFO:tasks.workunit.client.0.smithi186.stdout:./210/.snap/snap-subdir-test 2024-03-28T15:04:28.138 INFO:tasks.workunit.client.0.smithi186.stdout:./265/.snap/snap-subdir-test 2024-03-28T15:04:28.146 INFO:tasks.workunit.client.0.smithi186.stdout:./218/.snap/snap-subdir-test 2024-03-28T15:04:28.155 INFO:tasks.workunit.client.0.smithi186.stdout:./290/.snap/snap-subdir-test 2024-03-28T15:04:28.163 INFO:tasks.workunit.client.0.smithi186.stdout:./149/.snap/snap-subdir-test 2024-03-28T15:04:28.172 INFO:tasks.workunit.client.0.smithi186.stdout:./109/.snap/snap-subdir-test 2024-03-28T15:04:28.180 INFO:tasks.workunit.client.0.smithi186.stdout:./292/.snap/snap-subdir-test 2024-03-28T15:04:28.189 INFO:tasks.workunit.client.0.smithi186.stdout:./242/.snap/snap-subdir-test 2024-03-28T15:04:28.197 INFO:tasks.workunit.client.0.smithi186.stdout:./158/.snap/snap-subdir-test 2024-03-28T15:04:28.205 INFO:tasks.workunit.client.0.smithi186.stdout:./243/.snap/snap-subdir-test 2024-03-28T15:04:28.214 INFO:tasks.workunit.client.0.smithi186.stdout:./106/.snap/snap-subdir-test 2024-03-28T15:04:28.222 INFO:tasks.workunit.client.0.smithi186.stdout:./250/.snap/snap-subdir-test 2024-03-28T15:04:28.231 INFO:tasks.workunit.client.0.smithi186.stdout:./221/.snap/snap-subdir-test 2024-03-28T15:04:28.239 INFO:tasks.workunit.client.0.smithi186.stdout:./244/.snap/snap-subdir-test 2024-03-28T15:04:28.247 INFO:tasks.workunit.client.0.smithi186.stdout:./258/.snap/snap-subdir-test 2024-03-28T15:04:28.255 INFO:tasks.workunit.client.0.smithi186.stdout:./268/.snap/snap-subdir-test 2024-03-28T15:04:28.263 INFO:tasks.workunit.client.0.smithi186.stdout:./189/.snap/snap-subdir-test 2024-03-28T15:04:28.272 INFO:tasks.workunit.client.0.smithi186.stdout:./285/.snap/snap-subdir-test 2024-03-28T15:04:28.280 INFO:tasks.workunit.client.0.smithi186.stdout:./208/.snap/snap-subdir-test 2024-03-28T15:04:28.288 INFO:tasks.workunit.client.0.smithi186.stdout:./288/.snap/snap-subdir-test 2024-03-28T15:04:28.296 INFO:tasks.workunit.client.0.smithi186.stdout:./209/.snap/snap-subdir-test 2024-03-28T15:04:28.307 INFO:tasks.workunit.client.0.smithi186.stdout:./253/.snap/snap-subdir-test 2024-03-28T15:04:28.313 INFO:tasks.workunit.client.0.smithi186.stdout:./249/.snap/snap-subdir-test 2024-03-28T15:04:28.322 INFO:tasks.workunit.client.0.smithi186.stdout:./133/.snap/snap-subdir-test 2024-03-28T15:04:28.330 INFO:tasks.workunit.client.0.smithi186.stdout:./291/.snap/snap-subdir-test 2024-03-28T15:04:28.339 INFO:tasks.workunit.client.0.smithi186.stdout:./226/.snap/snap-subdir-test 2024-03-28T15:04:28.347 INFO:tasks.workunit.client.0.smithi186.stdout:./173/.snap/snap-subdir-test 2024-03-28T15:04:28.356 INFO:tasks.workunit.client.0.smithi186.stdout:./174/.snap/snap-subdir-test 2024-03-28T15:04:28.365 INFO:tasks.workunit.client.0.smithi186.stdout:./299/.snap/snap-subdir-test 2024-03-28T15:04:28.373 INFO:tasks.workunit.client.0.smithi186.stdout:./262/.snap/snap-subdir-test 2024-03-28T15:04:28.381 INFO:tasks.workunit.client.0.smithi186.stdout:./231/.snap/snap-subdir-test 2024-03-28T15:04:28.389 INFO:tasks.workunit.client.0.smithi186.stdout:./214/.snap/snap-subdir-test 2024-03-28T15:04:28.397 INFO:tasks.workunit.client.0.smithi186.stdout:./281/.snap/snap-subdir-test 2024-03-28T15:04:28.407 INFO:tasks.workunit.client.0.smithi186.stdout:./289/.snap/snap-subdir-test 2024-03-28T15:04:28.414 INFO:tasks.workunit.client.0.smithi186.stdout:./283/.snap/snap-subdir-test 2024-03-28T15:04:28.423 INFO:tasks.workunit.client.0.smithi186.stdout:./232/.snap/snap-subdir-test 2024-03-28T15:04:28.432 INFO:tasks.workunit.client.0.smithi186.stdout:./298/.snap/snap-subdir-test 2024-03-28T15:04:28.440 INFO:tasks.workunit.client.0.smithi186.stdout:./233/.snap/snap-subdir-test 2024-03-28T15:04:28.450 INFO:tasks.workunit.client.0.smithi186.stdout:./155/.snap/snap-subdir-test 2024-03-28T15:04:28.459 INFO:tasks.workunit.client.0.smithi186.stdout:./114/.snap/snap-subdir-test 2024-03-28T15:04:28.468 INFO:tasks.workunit.client.0.smithi186.stdout:./159/.snap/snap-subdir-test 2024-03-28T15:04:28.476 INFO:tasks.workunit.client.0.smithi186.stdout:./187/.snap/snap-subdir-test 2024-03-28T15:04:28.484 INFO:tasks.workunit.client.0.smithi186.stdout:./185/.snap/snap-subdir-test 2024-03-28T15:04:28.493 INFO:tasks.workunit.client.0.smithi186.stdout:./293/.snap/snap-subdir-test 2024-03-28T15:04:28.501 INFO:tasks.workunit.client.0.smithi186.stdout:./228/.snap/snap-subdir-test 2024-03-28T15:04:28.510 INFO:tasks.workunit.client.0.smithi186.stdout:./120/.snap/snap-subdir-test 2024-03-28T15:04:28.517 INFO:tasks.workunit.client.0.smithi186.stdout:./110/.snap/snap-subdir-test 2024-03-28T15:04:28.525 INFO:tasks.workunit.client.0.smithi186.stdout:./267/.snap/snap-subdir-test 2024-03-28T15:04:28.533 INFO:tasks.workunit.client.0.smithi186.stdout:./202/.snap/snap-subdir-test 2024-03-28T15:04:28.544 INFO:tasks.workunit.client.0.smithi186.stdout:./137/.snap/snap-subdir-test 2024-03-28T15:04:28.555 INFO:tasks.workunit.client.0.smithi186.stdout:./246/.snap/snap-subdir-test 2024-03-28T15:04:28.564 INFO:tasks.workunit.client.0.smithi186.stdout:./219/.snap/snap-subdir-test 2024-03-28T15:04:28.573 INFO:tasks.workunit.client.0.smithi186.stdout:./198/.snap/snap-subdir-test 2024-03-28T15:04:28.582 INFO:tasks.workunit.client.0.smithi186.stdout:./136/.snap/snap-subdir-test 2024-03-28T15:04:28.590 INFO:tasks.workunit.client.0.smithi186.stdout:./270/.snap/snap-subdir-test 2024-03-28T15:04:28.599 INFO:tasks.workunit.client.0.smithi186.stdout:./125/.snap/snap-subdir-test 2024-03-28T15:04:28.608 INFO:tasks.workunit.client.0.smithi186.stdout:./194/.snap/snap-subdir-test 2024-03-28T15:04:28.617 INFO:tasks.workunit.client.0.smithi186.stdout:./119/.snap/snap-subdir-test 2024-03-28T15:04:28.625 INFO:tasks.workunit.client.0.smithi186.stdout:./297/.snap/snap-subdir-test 2024-03-28T15:04:28.634 INFO:tasks.workunit.client.0.smithi186.stdout:./230/.snap/snap-subdir-test 2024-03-28T15:04:28.642 INFO:tasks.workunit.client.0.smithi186.stdout:./260/.snap/snap-subdir-test 2024-03-28T15:04:28.651 INFO:tasks.workunit.client.0.smithi186.stdout:./157/.snap/snap-subdir-test 2024-03-28T15:04:28.660 INFO:tasks.workunit.client.0.smithi186.stdout:./172/.snap/snap-subdir-test 2024-03-28T15:04:28.668 INFO:tasks.workunit.client.0.smithi186.stdout:./277/.snap/snap-subdir-test 2024-03-28T15:04:28.677 INFO:tasks.workunit.client.0.smithi186.stdout:./186/.snap/snap-subdir-test 2024-03-28T15:04:28.685 INFO:tasks.workunit.client.0.smithi186.stdout:./237/.snap/snap-subdir-test 2024-03-28T15:04:28.693 INFO:tasks.workunit.client.0.smithi186.stdout:./190/.snap/snap-subdir-test 2024-03-28T15:04:28.701 INFO:tasks.workunit.client.0.smithi186.stdout:./213/.snap/snap-subdir-test 2024-03-28T15:04:28.709 INFO:tasks.workunit.client.0.smithi186.stdout:./104/.snap/snap-subdir-test 2024-03-28T15:04:28.718 INFO:tasks.workunit.client.0.smithi186.stdout:./164/.snap/snap-subdir-test 2024-03-28T15:04:28.726 INFO:tasks.workunit.client.0.smithi186.stdout:./112/.snap/snap-subdir-test 2024-03-28T15:04:28.736 INFO:tasks.workunit.client.0.smithi186.stdout:./295/.snap/snap-subdir-test 2024-03-28T15:04:28.744 INFO:tasks.workunit.client.0.smithi186.stdout:./272/.snap/snap-subdir-test 2024-03-28T15:04:28.753 INFO:tasks.workunit.client.0.smithi186.stdout:./100/.snap/snap-subdir-test 2024-03-28T15:04:28.762 INFO:tasks.workunit.client.0.smithi186.stdout:./134/.snap/snap-subdir-test 2024-03-28T15:04:28.771 INFO:tasks.workunit.client.0.smithi186.stdout:./200/.snap/snap-subdir-test 2024-03-28T15:04:28.782 INFO:tasks.workunit.client.0.smithi186.stdout:./115/.snap/snap-subdir-test 2024-03-28T15:04:28.792 INFO:tasks.workunit.client.0.smithi186.stdout:./284/.snap/snap-subdir-test 2024-03-28T15:04:28.801 INFO:tasks.workunit.client.0.smithi186.stdout:./280/.snap/snap-subdir-test 2024-03-28T15:04:28.813 INFO:tasks.workunit.client.0.smithi186.stdout:./116/.snap/snap-subdir-test 2024-03-28T15:04:28.822 INFO:tasks.workunit.client.0.smithi186.stdout:./245/.snap/snap-subdir-test 2024-03-28T15:04:28.831 INFO:tasks.workunit.client.0.smithi186.stdout:./195/.snap/snap-subdir-test 2024-03-28T15:04:28.839 INFO:tasks.workunit.client.0.smithi186.stdout:./138/.snap/snap-subdir-test 2024-03-28T15:04:28.848 INFO:tasks.workunit.client.0.smithi186.stdout:./135/.snap/snap-subdir-test 2024-03-28T15:04:28.857 INFO:tasks.workunit.client.0.smithi186.stdout:./117/.snap/snap-subdir-test 2024-03-28T15:04:28.866 INFO:tasks.workunit.client.0.smithi186.stdout:./241/.snap/snap-subdir-test 2024-03-28T15:04:28.875 INFO:tasks.workunit.client.0.smithi186.stdout:./143/.snap/snap-subdir-test 2024-03-28T15:04:28.884 INFO:tasks.workunit.client.0.smithi186.stdout:./180/.snap/snap-subdir-test 2024-03-28T15:04:28.893 INFO:tasks.workunit.client.0.smithi186.stdout:./126/.snap/snap-subdir-test 2024-03-28T15:04:28.902 INFO:tasks.workunit.client.0.smithi186.stdout:./161/.snap/snap-subdir-test 2024-03-28T15:04:28.910 INFO:tasks.workunit.client.0.smithi186.stdout:./204/.snap/snap-subdir-test 2024-03-28T15:04:28.919 INFO:tasks.workunit.client.0.smithi186.stdout:./176/.snap/snap-subdir-test 2024-03-28T15:04:28.927 INFO:tasks.workunit.client.0.smithi186.stdout:./255/.snap/snap-subdir-test 2024-03-28T15:04:28.935 INFO:tasks.workunit.client.0.smithi186.stdout:./215/.snap/snap-subdir-test 2024-03-28T15:04:28.943 INFO:tasks.workunit.client.0.smithi186.stdout:./111/.snap/snap-subdir-test 2024-03-28T15:04:28.952 INFO:tasks.workunit.client.0.smithi186.stdout:./238/.snap/snap-subdir-test 2024-03-28T15:04:28.962 INFO:tasks.workunit.client.0.smithi186.stdout:./191/.snap/snap-subdir-test 2024-03-28T15:04:28.973 INFO:tasks.workunit.client.0.smithi186.stdout:./184/.snap/snap-subdir-test 2024-03-28T15:04:28.984 INFO:tasks.workunit.client.0.smithi186.stdout:./192/.snap/snap-subdir-test 2024-03-28T15:04:28.996 INFO:tasks.workunit.client.0.smithi186.stdout:./103/.snap/snap-subdir-test 2024-03-28T15:04:29.007 INFO:tasks.workunit.client.0.smithi186.stdout:./128/.snap/snap-subdir-test 2024-03-28T15:04:29.017 INFO:tasks.workunit.client.0.smithi186.stdout:./229/.snap/snap-subdir-test 2024-03-28T15:04:29.027 INFO:tasks.workunit.client.0.smithi186.stdout:./144/.snap/snap-subdir-test 2024-03-28T15:04:29.036 INFO:tasks.workunit.client.0.smithi186.stdout:./146/.snap/snap-subdir-test 2024-03-28T15:04:29.047 INFO:tasks.workunit.client.0.smithi186.stdout:./227/.snap/snap-subdir-test 2024-03-28T15:04:29.057 INFO:tasks.workunit.client.0.smithi186.stdout:./278/.snap/snap-subdir-test 2024-03-28T15:04:29.069 INFO:tasks.workunit.client.0.smithi186.stdout:./141/.snap/snap-subdir-test 2024-03-28T15:04:29.080 INFO:tasks.workunit.client.0.smithi186.stdout:./234/.snap/snap-subdir-test 2024-03-28T15:04:29.089 INFO:tasks.workunit.client.0.smithi186.stdout:./225/.snap/snap-subdir-test 2024-03-28T15:04:29.100 INFO:tasks.workunit.client.0.smithi186.stdout:./182/.snap/snap-subdir-test 2024-03-28T15:04:29.165 INFO:tasks.workunit.client.0.smithi186.stdout:./107/.snap/snap-subdir-test 2024-03-28T15:04:29.174 INFO:tasks.workunit.client.0.smithi186.stdout:./178/.snap/snap-subdir-test 2024-03-28T15:04:29.182 INFO:tasks.workunit.client.0.smithi186.stdout:./266/.snap/snap-subdir-test 2024-03-28T15:04:29.191 INFO:tasks.workunit.client.0.smithi186.stdout:./240/.snap/snap-subdir-test 2024-03-28T15:04:29.202 INFO:tasks.workunit.client.0.smithi186.stdout:./156/.snap/snap-subdir-test 2024-03-28T15:04:29.212 INFO:tasks.workunit.client.0.smithi186.stdout:./131/.snap/snap-subdir-test 2024-03-28T15:04:29.222 INFO:tasks.workunit.client.0.smithi186.stdout:./123/.snap/snap-subdir-test 2024-03-28T15:04:29.230 INFO:tasks.workunit.client.0.smithi186.stdout:./160/.snap/snap-subdir-test 2024-03-28T15:04:29.239 INFO:tasks.workunit.client.0.smithi186.stdout:./165/.snap/snap-subdir-test 2024-03-28T15:04:29.247 INFO:tasks.workunit.client.0.smithi186.stdout:./101/.snap/snap-subdir-test 2024-03-28T15:04:29.256 INFO:tasks.workunit.client.0.smithi186.stdout:./148/.snap/snap-subdir-test 2024-03-28T15:04:29.265 INFO:tasks.workunit.client.0.smithi186.stdout:./168/.snap/snap-subdir-test 2024-03-28T15:04:29.273 INFO:tasks.workunit.client.0.smithi186.stdout:./179/.snap/snap-subdir-test 2024-03-28T15:04:29.281 INFO:tasks.workunit.client.0.smithi186.stdout:./257/.snap/snap-subdir-test 2024-03-28T15:04:29.289 INFO:tasks.workunit.client.0.smithi186.stdout:./252/.snap/snap-subdir-test 2024-03-28T15:04:29.298 INFO:tasks.workunit.client.0.smithi186.stdout:./132/.snap/snap-subdir-test 2024-03-28T15:04:29.306 INFO:tasks.workunit.client.0.smithi186.stdout:./236/.snap/snap-subdir-test 2024-03-28T15:04:29.314 INFO:tasks.workunit.client.0.smithi186.stdout:./211/.snap/snap-subdir-test 2024-03-28T15:04:29.322 INFO:tasks.workunit.client.0.smithi186.stdout:./199/.snap/snap-subdir-test 2024-03-28T15:04:29.330 INFO:tasks.workunit.client.0.smithi186.stdout:./205/.snap/snap-subdir-test 2024-03-28T15:04:29.340 INFO:tasks.workunit.client.0.smithi186.stdout:./163/.snap/snap-subdir-test 2024-03-28T15:04:29.350 INFO:tasks.workunit.client.0.smithi186.stdout:./261/.snap/snap-subdir-test 2024-03-28T15:04:29.360 INFO:tasks.workunit.client.0.smithi186.stdout:./201/.snap/snap-subdir-test 2024-03-28T15:04:29.371 INFO:tasks.workunit.client.0.smithi186.stdout:./282/.snap/snap-subdir-test 2024-03-28T15:04:29.382 INFO:tasks.workunit.client.0.smithi186.stdout:./296/.snap/snap-subdir-test 2024-03-28T15:04:29.392 INFO:tasks.workunit.client.0.smithi186.stdout:./217/.snap/snap-subdir-test 2024-03-28T15:04:29.403 INFO:tasks.workunit.client.0.smithi186.stdout:./153/.snap/snap-subdir-test 2024-03-28T15:04:29.412 INFO:tasks.workunit.client.0.smithi186.stdout:./206/.snap/snap-subdir-test 2024-03-28T15:04:29.420 INFO:tasks.workunit.client.0.smithi186.stdout:./122/.snap/snap-subdir-test 2024-03-28T15:04:29.428 INFO:tasks.workunit.client.0.smithi186.stdout:./162/.snap/snap-subdir-test 2024-03-28T15:04:29.436 INFO:tasks.workunit.client.0.smithi186.stdout:./197/.snap/snap-subdir-test 2024-03-28T15:04:29.445 INFO:tasks.workunit.client.0.smithi186.stdout:./216/.snap/snap-subdir-test 2024-03-28T15:04:30.251 INFO:tasks.workunit.client.0.smithi186.stdout:./254/.snap/snap-subdir-test 2024-03-28T15:04:30.262 INFO:tasks.workunit.client.0.smithi186.stdout:./113/.snap/snap-subdir-test 2024-03-28T15:04:30.269 INFO:tasks.workunit.client.0.smithi186.stdout:./152/.snap/snap-subdir-test 2024-03-28T15:04:30.278 INFO:tasks.workunit.client.0.smithi186.stdout:./121/.snap/snap-subdir-test 2024-03-28T15:04:30.287 INFO:tasks.workunit.client.0.smithi186.stdout:./118/.snap/snap-subdir-test 2024-03-28T15:04:30.297 INFO:tasks.workunit.client.0.smithi186.stdout:./196/.snap/snap-subdir-test 2024-03-28T15:04:30.304 INFO:tasks.workunit.client.0.smithi186.stdout:./124/.snap/snap-subdir-test 2024-03-28T15:04:30.313 INFO:tasks.workunit.client.0.smithi186.stdout:./239/.snap/snap-subdir-test 2024-03-28T15:04:30.322 INFO:tasks.workunit.client.0.smithi186.stdout:./188/.snap/snap-subdir-test 2024-03-28T15:04:30.332 INFO:tasks.workunit.client.0.smithi186.stdout:./287/.snap/snap-subdir-test 2024-03-28T15:04:30.340 INFO:tasks.workunit.client.0.smithi186.stdout:./275/.snap/snap-subdir-test 2024-03-28T15:04:30.348 INFO:tasks.workunit.client.0.smithi186.stdout:./259/.snap/snap-subdir-test 2024-03-28T15:04:30.357 INFO:tasks.workunit.client.0.smithi186.stdout:./224/.snap/snap-subdir-test 2024-03-28T15:04:30.366 INFO:tasks.workunit.client.0.smithi186.stdout:./264/.snap/snap-subdir-test 2024-03-28T15:04:30.375 INFO:tasks.workunit.client.0.smithi186.stdout:./145/.snap/snap-subdir-test 2024-03-28T15:04:30.384 INFO:tasks.workunit.client.0.smithi186.stdout:./279/.snap/snap-subdir-test 2024-03-28T15:04:30.392 INFO:tasks.workunit.client.0.smithi186.stdout:./263/.snap/snap-subdir-test 2024-03-28T15:04:30.401 INFO:tasks.workunit.client.0.smithi186.stdout:./130/.snap/snap-subdir-test 2024-03-28T15:04:30.410 INFO:tasks.workunit.client.0.smithi186.stdout:./193/.snap/snap-subdir-test 2024-03-28T15:04:30.418 INFO:tasks.workunit.client.0.smithi186.stdout:./223/.snap/snap-subdir-test 2024-03-28T15:04:30.426 INFO:tasks.workunit.client.0.smithi186.stdout:./207/.snap/snap-subdir-test 2024-03-28T15:04:30.435 INFO:tasks.workunit.client.0.smithi186.stdout:./276/.snap/snap-subdir-test 2024-03-28T15:04:30.443 INFO:tasks.workunit.client.0.smithi186.stdout:./271/.snap/snap-subdir-test 2024-03-28T15:04:30.451 INFO:tasks.workunit.client.0.smithi186.stdout:./251/.snap/snap-subdir-test 2024-03-28T15:04:30.459 INFO:tasks.workunit.client.0.smithi186.stdout:./175/.snap/snap-subdir-test 2024-03-28T15:04:30.467 INFO:tasks.workunit.client.0.smithi186.stdout:./222/.snap/snap-subdir-test 2024-03-28T15:04:30.475 INFO:tasks.workunit.client.0.smithi186.stdout:./220/.snap/snap-subdir-test 2024-03-28T15:04:30.483 INFO:tasks.workunit.client.0.smithi186.stdout:./203/.snap/snap-subdir-test 2024-03-28T15:04:30.492 INFO:tasks.workunit.client.0.smithi186.stdout:./181/.snap/snap-subdir-test 2024-03-28T15:04:30.500 INFO:tasks.workunit.client.0.smithi186.stdout:./286/.snap/snap-subdir-test 2024-03-28T15:04:30.509 INFO:tasks.workunit.client.0.smithi186.stdout:./127/.snap/snap-subdir-test 2024-03-28T15:04:30.517 INFO:tasks.workunit.client.0.smithi186.stdout:./108/.snap/snap-subdir-test 2024-03-28T15:04:30.528 INFO:tasks.workunit.client.0.smithi186.stdout:./183/.snap/snap-subdir-test 2024-03-28T15:04:30.539 INFO:tasks.workunit.client.0.smithi186.stdout:./294/.snap/snap-subdir-test 2024-03-28T15:04:30.549 INFO:tasks.workunit.client.0.smithi186.stdout:./235/.snap/snap-subdir-test 2024-03-28T15:04:30.558 INFO:tasks.workunit.client.0.smithi186.stdout:./177/.snap/snap-subdir-test 2024-03-28T15:04:30.568 INFO:tasks.workunit.client.0.smithi186.stdout:./169/.snap/snap-subdir-test 2024-03-28T15:04:30.577 INFO:tasks.workunit.client.0.smithi186.stdout:./212/.snap/snap-subdir-test 2024-03-28T15:04:30.588 INFO:tasks.workunit.client.0.smithi186.stdout:./154/.snap/snap-subdir-test 2024-03-28T15:04:30.598 INFO:tasks.workunit.client.0.smithi186.stdout:./171/.snap/snap-subdir-test 2024-03-28T15:04:30.608 INFO:tasks.workunit.client.0.smithi186.stdout:./139/.snap/snap-subdir-test 2024-03-28T15:04:30.620 INFO:tasks.workunit.client.0.smithi186.stdout:./102/.snap/snap-subdir-test 2024-03-28T15:04:30.632 INFO:tasks.workunit.client.0.smithi186.stdout:./129/.snap/snap-subdir-test 2024-03-28T15:04:33.264 INFO:tasks.workunit.client.0.smithi186.stdout:Delete all the files and directories ... 2024-03-28T15:04:33.781 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/19' 2024-03-28T15:04:33.781 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/15' 2024-03-28T15:04:33.781 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/31' 2024-03-28T15:04:33.781 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/36' 2024-03-28T15:04:33.781 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/20' 2024-03-28T15:04:33.781 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/12' 2024-03-28T15:04:33.781 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/39' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/10' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/34' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/38' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/11' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/16' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/32' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/30' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/14' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/18' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/13' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/33' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/17' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/35' 2024-03-28T15:04:33.782 INFO:tasks.workunit.client.0.smithi186.stdout:removed './100/37' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './100' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/19' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/15' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/31' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/36' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/20' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/12' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/39' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/10' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/34' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/38' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/11' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/16' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/32' 2024-03-28T15:04:33.783 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/30' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/14' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/18' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/13' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/33' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/17' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/35' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './101/37' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './101' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/19' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/15' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/31' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/36' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/20' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/12' 2024-03-28T15:04:33.784 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/39' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/10' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/34' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/38' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/11' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/16' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/32' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/30' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/14' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/18' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/13' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/33' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/17' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/35' 2024-03-28T15:04:33.785 INFO:tasks.workunit.client.0.smithi186.stdout:removed './102/37' 2024-03-28T15:04:33.786 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './102' 2024-03-28T15:04:33.786 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/19' 2024-03-28T15:04:33.786 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/15' 2024-03-28T15:04:33.786 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/31' 2024-03-28T15:04:33.786 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/36' 2024-03-28T15:04:33.786 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/20' 2024-03-28T15:04:33.786 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/12' 2024-03-28T15:04:33.786 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/39' 2024-03-28T15:04:33.786 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/10' 2024-03-28T15:04:33.786 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/34' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/38' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/11' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/16' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/32' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/30' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/14' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/18' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/13' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/33' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/17' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/35' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './103/37' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './103' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/19' 2024-03-28T15:04:33.787 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/15' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/31' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/36' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/20' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/12' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/39' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/10' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/34' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/38' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/11' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/16' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/32' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/30' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/14' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/18' 2024-03-28T15:04:33.788 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/13' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/33' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/17' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/35' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './104/37' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './104' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/19' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/15' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/31' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/36' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/20' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/12' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/39' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/10' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/34' 2024-03-28T15:04:33.789 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/38' 2024-03-28T15:04:33.790 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/11' 2024-03-28T15:04:33.790 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/16' 2024-03-28T15:04:33.790 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/32' 2024-03-28T15:04:33.790 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/30' 2024-03-28T15:04:33.790 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/14' 2024-03-28T15:04:33.790 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/18' 2024-03-28T15:04:33.790 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/13' 2024-03-28T15:04:33.790 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/33' 2024-03-28T15:04:33.790 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/17' 2024-03-28T15:04:33.790 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/35' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './105/37' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './105' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/19' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/15' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/31' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/36' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/20' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/12' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/39' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/10' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/34' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/38' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/11' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/16' 2024-03-28T15:04:33.791 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/32' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/30' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/14' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/18' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/13' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/33' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/17' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/35' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './106/37' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './106' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/19' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/15' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/31' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/36' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/20' 2024-03-28T15:04:33.792 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/12' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/39' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/10' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/34' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/38' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/11' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/16' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/32' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/30' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/14' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/18' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/13' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/33' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/17' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/35' 2024-03-28T15:04:33.793 INFO:tasks.workunit.client.0.smithi186.stdout:removed './107/37' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './107' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/19' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/15' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/31' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/36' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/20' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/12' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/39' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/10' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/34' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/38' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/11' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/16' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/32' 2024-03-28T15:04:33.794 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/30' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/14' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/18' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/13' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/33' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/17' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/35' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './108/37' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './108' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/19' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/15' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/31' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/36' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/20' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/12' 2024-03-28T15:04:33.795 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/39' 2024-03-28T15:04:33.796 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/10' 2024-03-28T15:04:33.796 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/34' 2024-03-28T15:04:33.796 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/38' 2024-03-28T15:04:33.796 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/11' 2024-03-28T15:04:33.796 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/16' 2024-03-28T15:04:33.796 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/32' 2024-03-28T15:04:33.796 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/30' 2024-03-28T15:04:34.322 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/14' 2024-03-28T15:04:34.322 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/18' 2024-03-28T15:04:34.322 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/13' 2024-03-28T15:04:34.322 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/33' 2024-03-28T15:04:34.322 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/17' 2024-03-28T15:04:34.322 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/35' 2024-03-28T15:04:34.322 INFO:tasks.workunit.client.0.smithi186.stdout:removed './109/37' 2024-03-28T15:04:34.322 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './109' 2024-03-28T15:04:34.322 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/19' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/15' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/31' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/36' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/20' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/12' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/39' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/10' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/34' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/38' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/11' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/16' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/32' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/30' 2024-03-28T15:04:34.323 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/14' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/18' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/13' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/33' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/17' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/35' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed './110/37' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './110' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/19' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/15' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/31' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/36' 2024-03-28T15:04:34.324 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/20' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/12' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/39' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/10' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/34' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/38' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/11' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/16' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/32' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/30' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/14' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/18' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/13' 2024-03-28T15:04:34.325 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/33' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/17' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/35' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './111/37' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './111' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/19' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/15' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/31' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/36' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/20' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/12' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/39' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/10' 2024-03-28T15:04:34.326 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/34' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/38' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/11' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/16' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/32' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/30' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/14' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/18' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/13' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/33' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/17' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/35' 2024-03-28T15:04:34.327 INFO:tasks.workunit.client.0.smithi186.stdout:removed './112/37' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './112' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/19' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/15' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/31' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/36' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/20' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/12' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/39' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/10' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/34' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/38' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/11' 2024-03-28T15:04:34.328 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/16' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/32' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/30' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/14' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/18' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/13' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/33' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/17' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/35' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './113/37' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './113' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/19' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/15' 2024-03-28T15:04:34.329 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/31' 2024-03-28T15:04:34.330 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/36' 2024-03-28T15:04:34.330 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/20' 2024-03-28T15:04:34.330 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/12' 2024-03-28T15:04:34.330 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/39' 2024-03-28T15:04:34.330 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/10' 2024-03-28T15:04:34.330 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/34' 2024-03-28T15:04:34.330 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/38' 2024-03-28T15:04:34.330 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/11' 2024-03-28T15:04:34.330 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/16' 2024-03-28T15:04:34.330 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/32' 2024-03-28T15:04:34.330 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/30' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/14' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/18' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/13' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/33' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/17' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/35' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './114/37' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './114' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/19' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/15' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/31' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/36' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/20' 2024-03-28T15:04:34.331 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/12' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/39' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/10' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/34' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/38' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/11' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/16' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/32' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/30' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/14' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/18' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/13' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/33' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/17' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/35' 2024-03-28T15:04:34.332 INFO:tasks.workunit.client.0.smithi186.stdout:removed './115/37' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './115' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/19' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/15' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/31' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/36' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/20' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/12' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/39' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/10' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/34' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/38' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/11' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/16' 2024-03-28T15:04:34.333 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/32' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/30' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/14' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/18' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/13' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/33' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/17' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/35' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './116/37' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './116' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/19' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/15' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/31' 2024-03-28T15:04:34.334 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/36' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/20' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/12' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/39' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/10' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/34' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/38' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/11' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/16' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/32' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/30' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/14' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/18' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/13' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/33' 2024-03-28T15:04:34.335 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/17' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/35' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './117/37' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './117' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/19' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/15' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/31' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/36' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/20' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/12' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/39' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/10' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/34' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/38' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/11' 2024-03-28T15:04:34.336 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/16' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/32' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/30' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/14' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/18' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/13' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/33' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/17' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/35' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './118/37' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './118' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/19' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/15' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/31' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/36' 2024-03-28T15:04:34.337 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/20' 2024-03-28T15:04:34.338 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/12' 2024-03-28T15:04:34.849 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/39' 2024-03-28T15:04:34.849 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/10' 2024-03-28T15:04:34.849 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/34' 2024-03-28T15:04:34.849 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/38' 2024-03-28T15:04:34.849 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/11' 2024-03-28T15:04:34.849 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/16' 2024-03-28T15:04:34.849 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/32' 2024-03-28T15:04:34.849 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/30' 2024-03-28T15:04:34.849 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/14' 2024-03-28T15:04:34.849 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/18' 2024-03-28T15:04:34.849 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/13' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/33' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/17' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/35' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './119/37' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './119' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/19' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/15' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/31' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/36' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/20' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/12' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/39' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/10' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/34' 2024-03-28T15:04:34.850 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/38' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/11' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/16' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/32' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/30' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/14' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/18' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/13' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/33' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/17' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/35' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './120/37' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './120' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/19' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/15' 2024-03-28T15:04:34.851 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/31' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/36' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/20' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/12' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/39' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/10' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/34' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/38' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/11' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/16' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/32' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/30' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/14' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/18' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/13' 2024-03-28T15:04:34.852 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/33' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/17' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/35' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './121/37' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './121' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/19' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/15' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/31' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/36' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/20' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/12' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/39' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/10' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/34' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/38' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/11' 2024-03-28T15:04:34.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/16' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/32' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/30' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/14' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/18' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/13' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/33' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/17' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/35' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './122/37' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './122' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/19' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/15' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/31' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/36' 2024-03-28T15:04:34.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/20' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/12' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/39' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/10' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/34' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/38' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/11' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/16' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/32' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/30' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/14' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/18' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/13' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/33' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/17' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/35' 2024-03-28T15:04:34.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './123/37' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './123' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/19' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/15' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/31' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/36' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/20' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/12' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/39' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/10' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/34' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/38' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/11' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/16' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/32' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/30' 2024-03-28T15:04:34.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/14' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/18' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/13' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/33' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/17' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/35' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './124/37' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './124' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/19' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/15' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/31' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/36' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/20' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/12' 2024-03-28T15:04:34.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/39' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/10' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/34' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/38' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/11' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/16' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/32' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/30' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/14' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/18' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/13' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/33' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/17' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/35' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './125/37' 2024-03-28T15:04:34.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './125' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/19' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/15' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/31' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/36' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/20' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/12' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/39' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/10' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/34' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/38' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/11' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/16' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/32' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/30' 2024-03-28T15:04:34.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/14' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/18' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/13' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/33' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/17' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/35' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './126/37' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './126' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/19' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/15' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/31' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/36' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/20' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/12' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/39' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/10' 2024-03-28T15:04:34.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/34' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/38' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/11' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/16' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/32' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/30' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/14' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/18' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/13' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/33' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/17' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/35' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './127/37' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './127' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/19' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/15' 2024-03-28T15:04:34.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/31' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/36' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/20' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/12' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/39' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/10' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/34' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/38' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/11' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/16' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/32' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/30' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/14' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/18' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/13' 2024-03-28T15:04:34.862 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/33' 2024-03-28T15:04:34.863 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/17' 2024-03-28T15:04:34.863 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/35' 2024-03-28T15:04:35.382 INFO:tasks.workunit.client.0.smithi186.stdout:removed './128/37' 2024-03-28T15:04:35.382 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './128' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/19' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/15' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/31' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/36' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/20' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/12' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/39' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/10' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/34' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/38' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/11' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/16' 2024-03-28T15:04:35.383 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/32' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/30' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/14' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/18' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/13' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/33' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/17' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/35' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed './129/37' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './129' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/19' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/15' 2024-03-28T15:04:35.384 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/31' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/36' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/20' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/12' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/39' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/10' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/34' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/38' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/11' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/16' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/32' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/30' 2024-03-28T15:04:35.385 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/14' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/18' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/13' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/33' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/17' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/35' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed './130/37' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './130' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/19' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/15' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/31' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/36' 2024-03-28T15:04:35.386 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/20' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/12' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/39' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/10' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/34' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/38' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/11' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/16' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/32' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/30' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/14' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/18' 2024-03-28T15:04:35.387 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/13' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/33' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/17' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/35' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed './131/37' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './131' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/19' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/15' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/31' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/36' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/20' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/12' 2024-03-28T15:04:35.388 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/39' 2024-03-28T15:04:35.389 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/10' 2024-03-28T15:04:35.389 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/34' 2024-03-28T15:04:35.389 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/38' 2024-03-28T15:04:35.389 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/11' 2024-03-28T15:04:35.389 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/16' 2024-03-28T15:04:35.389 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/32' 2024-03-28T15:04:35.389 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/30' 2024-03-28T15:04:35.389 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/14' 2024-03-28T15:04:35.389 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/18' 2024-03-28T15:04:35.389 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/13' 2024-03-28T15:04:35.389 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/33' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/17' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/35' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './132/37' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './132' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/19' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/15' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/31' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/36' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/20' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/12' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/39' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/10' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/34' 2024-03-28T15:04:35.390 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/38' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/11' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/16' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/32' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/30' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/14' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/18' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/13' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/33' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/17' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/35' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './133/37' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './133' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/19' 2024-03-28T15:04:35.391 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/15' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/31' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/36' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/20' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/12' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/39' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/10' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/34' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/38' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/11' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/16' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/32' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/30' 2024-03-28T15:04:35.392 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/14' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/18' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/13' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/33' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/17' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/35' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed './134/37' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './134' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/19' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/15' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/31' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/36' 2024-03-28T15:04:35.393 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/20' 2024-03-28T15:04:35.394 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/12' 2024-03-28T15:04:35.394 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/39' 2024-03-28T15:04:35.394 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/10' 2024-03-28T15:04:35.394 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/34' 2024-03-28T15:04:35.394 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/38' 2024-03-28T15:04:35.394 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/11' 2024-03-28T15:04:35.394 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/16' 2024-03-28T15:04:35.394 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/32' 2024-03-28T15:04:35.394 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/30' 2024-03-28T15:04:35.394 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/14' 2024-03-28T15:04:35.395 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/18' 2024-03-28T15:04:35.395 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/13' 2024-03-28T15:04:35.395 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/33' 2024-03-28T15:04:35.395 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/17' 2024-03-28T15:04:35.395 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/35' 2024-03-28T15:04:35.395 INFO:tasks.workunit.client.0.smithi186.stdout:removed './135/37' 2024-03-28T15:04:35.395 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './135' 2024-03-28T15:04:35.395 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/19' 2024-03-28T15:04:35.395 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/15' 2024-03-28T15:04:35.395 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/31' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/36' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/20' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/12' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/39' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/10' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/34' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/38' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/11' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/16' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/32' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/30' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/14' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/18' 2024-03-28T15:04:35.396 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/13' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/33' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/17' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/35' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed './136/37' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './136' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/19' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/15' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/31' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/36' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/20' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/12' 2024-03-28T15:04:35.397 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/39' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/10' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/34' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/38' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/11' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/16' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/32' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/30' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/14' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/18' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/13' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/33' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/17' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/35' 2024-03-28T15:04:35.398 INFO:tasks.workunit.client.0.smithi186.stdout:removed './137/37' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './137' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/19' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/15' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/31' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/36' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/20' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/12' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/39' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/10' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/34' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/38' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/11' 2024-03-28T15:04:35.399 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/16' 2024-03-28T15:04:36.344 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/32' 2024-03-28T15:04:36.344 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/30' 2024-03-28T15:04:36.344 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/14' 2024-03-28T15:04:36.344 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/18' 2024-03-28T15:04:36.344 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/13' 2024-03-28T15:04:36.345 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/33' 2024-03-28T15:04:36.345 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/17' 2024-03-28T15:04:36.345 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/35' 2024-03-28T15:04:36.345 INFO:tasks.workunit.client.0.smithi186.stdout:removed './138/37' 2024-03-28T15:04:36.345 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './138' 2024-03-28T15:04:36.345 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/19' 2024-03-28T15:04:36.345 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/15' 2024-03-28T15:04:36.346 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/31' 2024-03-28T15:04:36.346 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/36' 2024-03-28T15:04:36.346 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/20' 2024-03-28T15:04:36.346 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/12' 2024-03-28T15:04:36.346 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/39' 2024-03-28T15:04:36.346 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/10' 2024-03-28T15:04:36.346 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/34' 2024-03-28T15:04:36.347 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/38' 2024-03-28T15:04:36.347 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/11' 2024-03-28T15:04:36.347 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/16' 2024-03-28T15:04:36.347 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/32' 2024-03-28T15:04:36.347 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/30' 2024-03-28T15:04:36.347 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/14' 2024-03-28T15:04:36.347 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/18' 2024-03-28T15:04:36.347 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/13' 2024-03-28T15:04:36.348 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/33' 2024-03-28T15:04:36.348 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/17' 2024-03-28T15:04:36.348 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/35' 2024-03-28T15:04:36.348 INFO:tasks.workunit.client.0.smithi186.stdout:removed './139/37' 2024-03-28T15:04:36.348 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './139' 2024-03-28T15:04:36.348 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/19' 2024-03-28T15:04:36.348 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/15' 2024-03-28T15:04:36.348 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/31' 2024-03-28T15:04:36.348 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/36' 2024-03-28T15:04:36.349 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/20' 2024-03-28T15:04:36.349 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/12' 2024-03-28T15:04:36.349 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/39' 2024-03-28T15:04:36.349 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/10' 2024-03-28T15:04:36.349 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/34' 2024-03-28T15:04:36.349 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/38' 2024-03-28T15:04:36.349 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/11' 2024-03-28T15:04:36.349 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/16' 2024-03-28T15:04:36.350 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/32' 2024-03-28T15:04:36.350 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/30' 2024-03-28T15:04:36.350 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/14' 2024-03-28T15:04:36.350 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/18' 2024-03-28T15:04:36.350 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/13' 2024-03-28T15:04:36.350 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/33' 2024-03-28T15:04:36.350 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/17' 2024-03-28T15:04:36.350 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/35' 2024-03-28T15:04:36.350 INFO:tasks.workunit.client.0.smithi186.stdout:removed './140/37' 2024-03-28T15:04:36.350 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './140' 2024-03-28T15:04:36.351 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/19' 2024-03-28T15:04:36.351 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/15' 2024-03-28T15:04:36.351 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/31' 2024-03-28T15:04:36.351 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/36' 2024-03-28T15:04:36.351 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/20' 2024-03-28T15:04:36.351 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/12' 2024-03-28T15:04:36.351 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/39' 2024-03-28T15:04:36.351 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/10' 2024-03-28T15:04:36.351 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/34' 2024-03-28T15:04:36.352 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/38' 2024-03-28T15:04:36.352 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/11' 2024-03-28T15:04:36.352 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/16' 2024-03-28T15:04:36.352 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/32' 2024-03-28T15:04:36.352 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/30' 2024-03-28T15:04:36.352 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/14' 2024-03-28T15:04:36.352 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/18' 2024-03-28T15:04:36.352 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/13' 2024-03-28T15:04:36.352 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/33' 2024-03-28T15:04:36.352 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/17' 2024-03-28T15:04:36.352 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/35' 2024-03-28T15:04:36.353 INFO:tasks.workunit.client.0.smithi186.stdout:removed './141/37' 2024-03-28T15:04:36.353 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './141' 2024-03-28T15:04:36.353 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/19' 2024-03-28T15:04:36.353 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/15' 2024-03-28T15:04:36.353 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/31' 2024-03-28T15:04:36.353 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/36' 2024-03-28T15:04:36.353 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/20' 2024-03-28T15:04:36.353 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/12' 2024-03-28T15:04:36.353 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/39' 2024-03-28T15:04:36.353 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/10' 2024-03-28T15:04:36.353 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/34' 2024-03-28T15:04:36.354 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/38' 2024-03-28T15:04:36.354 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/11' 2024-03-28T15:04:36.354 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/16' 2024-03-28T15:04:36.354 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/32' 2024-03-28T15:04:36.354 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/30' 2024-03-28T15:04:36.354 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/14' 2024-03-28T15:04:36.354 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/18' 2024-03-28T15:04:36.354 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/13' 2024-03-28T15:04:36.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/33' 2024-03-28T15:04:36.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/17' 2024-03-28T15:04:36.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/35' 2024-03-28T15:04:36.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './142/37' 2024-03-28T15:04:36.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './142' 2024-03-28T15:04:36.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/19' 2024-03-28T15:04:36.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/15' 2024-03-28T15:04:36.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/31' 2024-03-28T15:04:36.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/36' 2024-03-28T15:04:36.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/20' 2024-03-28T15:04:36.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/12' 2024-03-28T15:04:36.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/39' 2024-03-28T15:04:36.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/10' 2024-03-28T15:04:36.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/34' 2024-03-28T15:04:36.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/38' 2024-03-28T15:04:36.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/11' 2024-03-28T15:04:36.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/16' 2024-03-28T15:04:36.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/32' 2024-03-28T15:04:36.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/30' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/14' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/18' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/13' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/33' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/17' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/35' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './143/37' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './143' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/19' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/15' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/31' 2024-03-28T15:04:36.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/36' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/20' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/12' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/39' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/10' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/34' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/38' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/11' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/16' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/32' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/30' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/14' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/18' 2024-03-28T15:04:36.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/13' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/33' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/17' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/35' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './144/37' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './144' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/19' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/15' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/31' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/36' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/20' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/12' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/39' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/10' 2024-03-28T15:04:36.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/34' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/38' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/11' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/16' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/32' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/30' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/14' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/18' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/13' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/33' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/17' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/35' 2024-03-28T15:04:36.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './145/37' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './145' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/19' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/15' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/31' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/36' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/20' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/12' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/39' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/10' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/34' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/38' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/11' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/16' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/32' 2024-03-28T15:04:36.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/30' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/14' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/18' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/13' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/33' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/17' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/35' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './146/37' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './146' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/19' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/15' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/31' 2024-03-28T15:04:36.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/36' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/20' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/12' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/39' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/10' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/34' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/38' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/11' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/16' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/32' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/30' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/14' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/18' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/13' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/33' 2024-03-28T15:04:36.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/17' 2024-03-28T15:04:36.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/35' 2024-03-28T15:04:36.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './147/37' 2024-03-28T15:04:36.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './147' 2024-03-28T15:04:36.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/19' 2024-03-28T15:04:36.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/15' 2024-03-28T15:04:36.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/31' 2024-03-28T15:04:36.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/36' 2024-03-28T15:04:37.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/20' 2024-03-28T15:04:37.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/12' 2024-03-28T15:04:37.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/39' 2024-03-28T15:04:37.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/10' 2024-03-28T15:04:37.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/34' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/38' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/11' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/16' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/32' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/30' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/14' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/18' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/13' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/33' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/17' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/35' 2024-03-28T15:04:37.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './148/37' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './148' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/19' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/15' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/31' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/36' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/20' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/12' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/39' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/10' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/34' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/38' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/11' 2024-03-28T15:04:37.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/16' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/32' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/30' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/14' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/18' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/13' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/33' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/17' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/35' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './149/37' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './149' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/19' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/15' 2024-03-28T15:04:37.483 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/31' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/36' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/20' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/12' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/39' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/10' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/34' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/38' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/11' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/16' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/32' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/30' 2024-03-28T15:04:37.484 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/14' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/18' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/13' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/33' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/17' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/35' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './150/37' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './150' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/19' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/15' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/31' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/36' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/20' 2024-03-28T15:04:37.485 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/12' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/39' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/10' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/34' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/38' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/11' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/16' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/32' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/30' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/14' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/18' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/13' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/33' 2024-03-28T15:04:37.486 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/17' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/35' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './151/37' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './151' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/19' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/15' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/31' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/36' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/20' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/12' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/39' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/10' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/34' 2024-03-28T15:04:37.487 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/38' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/11' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/16' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/32' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/30' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/14' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/18' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/13' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/33' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/17' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/35' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed './152/37' 2024-03-28T15:04:37.488 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './152' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/19' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/15' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/31' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/36' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/20' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/12' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/39' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/10' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/34' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/38' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/11' 2024-03-28T15:04:37.489 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/16' 2024-03-28T15:04:37.490 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/32' 2024-03-28T15:04:37.490 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/30' 2024-03-28T15:04:37.490 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/14' 2024-03-28T15:04:37.490 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/18' 2024-03-28T15:04:37.490 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/13' 2024-03-28T15:04:37.490 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/33' 2024-03-28T15:04:37.490 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/17' 2024-03-28T15:04:37.490 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/35' 2024-03-28T15:04:37.490 INFO:tasks.workunit.client.0.smithi186.stdout:removed './153/37' 2024-03-28T15:04:37.491 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './153' 2024-03-28T15:04:37.491 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/19' 2024-03-28T15:04:37.491 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/15' 2024-03-28T15:04:37.491 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/31' 2024-03-28T15:04:37.491 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/36' 2024-03-28T15:04:37.491 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/20' 2024-03-28T15:04:37.491 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/12' 2024-03-28T15:04:37.491 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/39' 2024-03-28T15:04:37.491 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/10' 2024-03-28T15:04:37.491 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/34' 2024-03-28T15:04:37.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/38' 2024-03-28T15:04:37.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/11' 2024-03-28T15:04:37.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/16' 2024-03-28T15:04:37.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/32' 2024-03-28T15:04:37.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/30' 2024-03-28T15:04:37.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/14' 2024-03-28T15:04:37.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/18' 2024-03-28T15:04:37.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/13' 2024-03-28T15:04:37.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/33' 2024-03-28T15:04:37.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/17' 2024-03-28T15:04:37.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/35' 2024-03-28T15:04:37.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './154/37' 2024-03-28T15:04:37.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './154' 2024-03-28T15:04:37.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/19' 2024-03-28T15:04:37.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/15' 2024-03-28T15:04:37.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/31' 2024-03-28T15:04:37.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/36' 2024-03-28T15:04:37.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/20' 2024-03-28T15:04:37.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/12' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/39' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/10' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/34' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/38' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/11' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/16' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/32' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/30' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/14' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/18' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/13' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/33' 2024-03-28T15:04:37.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/17' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/35' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './155/37' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './155' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/19' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/15' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/31' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/36' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/20' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/12' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/39' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/10' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/34' 2024-03-28T15:04:37.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/38' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/11' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/16' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/32' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/30' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/14' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/18' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/13' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/33' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/17' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/35' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './156/37' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './156' 2024-03-28T15:04:37.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/19' 2024-03-28T15:04:37.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/15' 2024-03-28T15:04:37.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/31' 2024-03-28T15:04:37.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/36' 2024-03-28T15:04:37.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/20' 2024-03-28T15:04:37.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/12' 2024-03-28T15:04:37.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/39' 2024-03-28T15:04:37.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/10' 2024-03-28T15:04:37.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/34' 2024-03-28T15:04:37.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/38' 2024-03-28T15:04:37.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/11' 2024-03-28T15:04:37.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/16' 2024-03-28T15:04:37.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/32' 2024-03-28T15:04:37.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/30' 2024-03-28T15:04:37.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/14' 2024-03-28T15:04:37.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/18' 2024-03-28T15:04:37.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/13' 2024-03-28T15:04:37.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/33' 2024-03-28T15:04:38.056 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/17' 2024-03-28T15:04:38.056 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/35' 2024-03-28T15:04:38.056 INFO:tasks.workunit.client.0.smithi186.stdout:removed './157/37' 2024-03-28T15:04:38.056 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './157' 2024-03-28T15:04:38.056 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/19' 2024-03-28T15:04:38.056 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/15' 2024-03-28T15:04:38.056 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/31' 2024-03-28T15:04:38.056 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/36' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/20' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/12' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/39' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/10' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/34' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/38' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/11' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/16' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/32' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/30' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/14' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/18' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/13' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/33' 2024-03-28T15:04:38.057 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/17' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/35' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './158/37' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './158' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/19' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/15' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/31' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/36' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/20' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/12' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/39' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/10' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/34' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/38' 2024-03-28T15:04:38.058 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/11' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/16' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/32' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/30' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/14' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/18' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/13' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/33' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/17' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/35' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './159/37' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './159' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/19' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/15' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/31' 2024-03-28T15:04:38.059 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/36' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/20' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/12' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/39' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/10' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/34' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/38' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/11' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/16' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/32' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/30' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/14' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/18' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/13' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/33' 2024-03-28T15:04:38.060 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/17' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/35' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './160/37' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './160' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/19' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/15' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/31' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/36' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/20' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/12' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/39' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/10' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/34' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/38' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/11' 2024-03-28T15:04:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/16' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/32' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/30' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/14' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/18' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/13' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/33' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/17' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/35' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './161/37' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './161' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/19' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/15' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/31' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/36' 2024-03-28T15:04:38.062 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/20' 2024-03-28T15:04:38.063 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/12' 2024-03-28T15:04:38.063 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/39' 2024-03-28T15:04:38.063 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/10' 2024-03-28T15:04:38.063 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/34' 2024-03-28T15:04:38.063 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/38' 2024-03-28T15:04:38.063 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/11' 2024-03-28T15:04:38.063 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/16' 2024-03-28T15:04:38.063 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/32' 2024-03-28T15:04:38.063 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/30' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/14' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/18' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/13' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/33' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/17' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/35' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './162/37' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './162' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/19' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/15' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/31' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/36' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/20' 2024-03-28T15:04:38.064 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/12' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/39' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/10' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/34' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/38' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/11' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/16' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/32' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/30' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/14' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/18' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/13' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/33' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/17' 2024-03-28T15:04:38.065 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/35' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './163/37' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './163' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/19' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/15' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/31' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/36' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/20' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/12' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/39' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/10' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/34' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/38' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/11' 2024-03-28T15:04:38.066 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/16' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/32' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/30' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/14' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/18' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/13' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/33' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/17' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/35' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './164/37' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './164' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/19' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/15' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/31' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/36' 2024-03-28T15:04:38.067 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/20' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/12' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/39' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/10' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/34' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/38' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/11' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/16' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/32' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/30' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/14' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/18' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/13' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/33' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/17' 2024-03-28T15:04:38.068 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/35' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './165/37' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './165' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/19' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/15' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/31' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/36' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/20' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/12' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/39' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/10' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/34' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/38' 2024-03-28T15:04:38.069 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/11' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/16' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/32' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/30' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/14' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/18' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/13' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/33' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/17' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/35' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './166/37' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './166' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/19' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/15' 2024-03-28T15:04:38.070 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/31' 2024-03-28T15:04:38.071 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/36' 2024-03-28T15:04:38.071 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/20' 2024-03-28T15:04:38.071 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/12' 2024-03-28T15:04:38.071 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/39' 2024-03-28T15:04:38.071 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/10' 2024-03-28T15:04:38.071 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/34' 2024-03-28T15:04:38.071 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/38' 2024-03-28T15:04:38.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/11' 2024-03-28T15:04:38.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/16' 2024-03-28T15:04:38.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/32' 2024-03-28T15:04:38.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/30' 2024-03-28T15:04:38.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/14' 2024-03-28T15:04:38.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/18' 2024-03-28T15:04:38.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/13' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/33' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/17' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/35' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './167/37' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './167' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/19' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/15' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/31' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/36' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/20' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/12' 2024-03-28T15:04:38.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/39' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/10' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/34' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/38' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/11' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/16' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/32' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/30' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/14' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/18' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/13' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/33' 2024-03-28T15:04:38.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/17' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/35' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './168/37' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './168' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/19' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/15' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/31' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/36' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/20' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/12' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/39' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/10' 2024-03-28T15:04:38.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/34' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/38' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/11' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/16' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/32' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/30' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/14' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/18' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/13' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/33' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/17' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/35' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './169/37' 2024-03-28T15:04:38.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './169' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/19' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/15' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/31' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/36' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/20' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/12' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/39' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/10' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/34' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/38' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/11' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/16' 2024-03-28T15:04:38.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/32' 2024-03-28T15:04:38.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/30' 2024-03-28T15:04:38.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/14' 2024-03-28T15:04:38.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/18' 2024-03-28T15:04:38.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/13' 2024-03-28T15:04:38.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/33' 2024-03-28T15:04:38.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/17' 2024-03-28T15:04:38.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/35' 2024-03-28T15:04:38.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './170/37' 2024-03-28T15:04:38.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './170' 2024-03-28T15:04:38.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/19' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/15' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/31' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/36' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/20' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/12' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/39' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/10' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/34' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/38' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/11' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/16' 2024-03-28T15:04:38.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/32' 2024-03-28T15:04:38.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/30' 2024-03-28T15:04:38.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/14' 2024-03-28T15:04:38.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/18' 2024-03-28T15:04:38.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/13' 2024-03-28T15:04:38.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/33' 2024-03-28T15:04:38.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/17' 2024-03-28T15:04:38.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/35' 2024-03-28T15:04:38.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './171/37' 2024-03-28T15:04:38.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './171' 2024-03-28T15:04:38.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/19' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/15' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/31' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/36' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/20' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/12' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/39' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/10' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/34' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/38' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/11' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/16' 2024-03-28T15:04:38.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/32' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/30' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/14' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/18' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/13' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/33' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/17' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/35' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './172/37' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './172' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/19' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/15' 2024-03-28T15:04:38.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/31' 2024-03-28T15:04:38.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/36' 2024-03-28T15:04:38.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/20' 2024-03-28T15:04:38.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/12' 2024-03-28T15:04:38.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/39' 2024-03-28T15:04:38.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/10' 2024-03-28T15:04:38.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/34' 2024-03-28T15:04:38.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/38' 2024-03-28T15:04:38.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/11' 2024-03-28T15:04:38.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/16' 2024-03-28T15:04:38.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/32' 2024-03-28T15:04:38.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/30' 2024-03-28T15:04:38.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/14' 2024-03-28T15:04:38.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/18' 2024-03-28T15:04:38.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/13' 2024-03-28T15:04:38.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/33' 2024-03-28T15:04:38.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/17' 2024-03-28T15:04:38.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/35' 2024-03-28T15:04:38.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './173/37' 2024-03-28T15:04:38.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './173' 2024-03-28T15:04:38.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/19' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/15' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/31' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/36' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/20' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/12' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/39' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/10' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/34' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/38' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/11' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/16' 2024-03-28T15:04:38.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/32' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/30' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/14' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/18' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/13' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/33' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/17' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/35' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './174/37' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './174' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/19' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/15' 2024-03-28T15:04:38.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/31' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/36' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/20' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/12' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/39' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/10' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/34' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/38' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/11' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/16' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/32' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/30' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/14' 2024-03-28T15:04:38.663 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/18' 2024-03-28T15:04:38.664 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/13' 2024-03-28T15:04:38.664 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/33' 2024-03-28T15:04:38.664 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/17' 2024-03-28T15:04:38.664 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/35' 2024-03-28T15:04:38.664 INFO:tasks.workunit.client.0.smithi186.stdout:removed './175/37' 2024-03-28T15:04:38.664 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './175' 2024-03-28T15:04:38.664 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/19' 2024-03-28T15:04:38.664 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/15' 2024-03-28T15:04:38.664 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/31' 2024-03-28T15:04:38.664 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/36' 2024-03-28T15:04:38.664 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/20' 2024-03-28T15:04:38.665 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/12' 2024-03-28T15:04:38.665 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/39' 2024-03-28T15:04:38.665 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/10' 2024-03-28T15:04:38.665 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/34' 2024-03-28T15:04:38.665 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/38' 2024-03-28T15:04:38.665 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/11' 2024-03-28T15:04:38.665 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/16' 2024-03-28T15:04:38.665 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/32' 2024-03-28T15:04:38.665 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/30' 2024-03-28T15:04:38.666 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/14' 2024-03-28T15:04:38.666 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/18' 2024-03-28T15:04:38.666 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/13' 2024-03-28T15:04:38.666 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/33' 2024-03-28T15:04:38.666 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/17' 2024-03-28T15:04:38.666 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/35' 2024-03-28T15:04:38.666 INFO:tasks.workunit.client.0.smithi186.stdout:removed './176/37' 2024-03-28T15:04:38.666 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './176' 2024-03-28T15:04:38.667 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/19' 2024-03-28T15:04:38.667 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/15' 2024-03-28T15:04:39.102 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/31' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/36' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/20' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/12' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/39' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/10' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/34' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/38' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/11' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/16' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/32' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/30' 2024-03-28T15:04:39.103 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/14' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/18' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/13' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/33' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/17' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/35' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './177/37' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './177' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/19' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/15' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/31' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/36' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/20' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/12' 2024-03-28T15:04:39.104 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/39' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/10' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/34' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/38' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/11' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/16' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/32' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/30' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/14' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/18' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/13' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/33' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/17' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/35' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed './178/37' 2024-03-28T15:04:39.105 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './178' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/19' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/15' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/31' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/36' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/20' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/12' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/39' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/10' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/34' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/38' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/11' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/16' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/32' 2024-03-28T15:04:39.106 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/30' 2024-03-28T15:04:39.107 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/14' 2024-03-28T15:04:39.107 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/18' 2024-03-28T15:04:39.107 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/13' 2024-03-28T15:04:39.107 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/33' 2024-03-28T15:04:39.107 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/17' 2024-03-28T15:04:39.107 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/35' 2024-03-28T15:04:39.107 INFO:tasks.workunit.client.0.smithi186.stdout:removed './179/37' 2024-03-28T15:04:39.107 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './179' 2024-03-28T15:04:39.107 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/19' 2024-03-28T15:04:39.107 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/15' 2024-03-28T15:04:39.107 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/31' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/36' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/20' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/12' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/39' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/10' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/34' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/38' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/11' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/16' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/32' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/30' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/14' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/18' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/13' 2024-03-28T15:04:39.108 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/33' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/17' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/35' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './180/37' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './180' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/19' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/15' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/31' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/36' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/20' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/12' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/39' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/10' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/34' 2024-03-28T15:04:39.109 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/38' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/11' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/16' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/32' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/30' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/14' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/18' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/13' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/33' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/17' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/35' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './181/37' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './181' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/19' 2024-03-28T15:04:39.110 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/15' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/31' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/36' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/20' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/12' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/39' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/10' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/34' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/38' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/11' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/16' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/32' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/30' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/14' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/18' 2024-03-28T15:04:39.111 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/13' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/33' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/17' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/35' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './182/37' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './182' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/19' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/15' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/31' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/36' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/20' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/12' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/39' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/10' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/34' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/38' 2024-03-28T15:04:39.112 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/11' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/16' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/32' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/30' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/14' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/18' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/13' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/33' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/17' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/35' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './183/37' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './183' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/19' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/15' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/31' 2024-03-28T15:04:39.113 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/36' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/20' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/12' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/39' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/10' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/34' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/38' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/11' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/16' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/32' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/30' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/14' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/18' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/13' 2024-03-28T15:04:39.114 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/33' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/17' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/35' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './184/37' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './184' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/19' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/15' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/31' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/36' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/20' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/12' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/39' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/10' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/34' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/38' 2024-03-28T15:04:39.115 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/11' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/16' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/32' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/30' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/14' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/18' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/13' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/33' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/17' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/35' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './185/37' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './185' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/19' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/15' 2024-03-28T15:04:39.116 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/31' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/36' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/20' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/12' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/39' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/10' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/34' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/38' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/11' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/16' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/32' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/30' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/14' 2024-03-28T15:04:39.117 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/18' 2024-03-28T15:04:39.738 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/13' 2024-03-28T15:04:39.738 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/33' 2024-03-28T15:04:39.738 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/17' 2024-03-28T15:04:39.738 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/35' 2024-03-28T15:04:39.738 INFO:tasks.workunit.client.0.smithi186.stdout:removed './186/37' 2024-03-28T15:04:39.738 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './186' 2024-03-28T15:04:39.738 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/19' 2024-03-28T15:04:39.738 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/15' 2024-03-28T15:04:39.738 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/31' 2024-03-28T15:04:39.739 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/36' 2024-03-28T15:04:39.739 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/20' 2024-03-28T15:04:39.739 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/12' 2024-03-28T15:04:39.739 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/39' 2024-03-28T15:04:39.739 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/10' 2024-03-28T15:04:39.739 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/34' 2024-03-28T15:04:39.739 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/38' 2024-03-28T15:04:39.739 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/11' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/16' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/32' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/30' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/14' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/18' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/13' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/33' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/17' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/35' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './187/37' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './187' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/19' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/15' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/31' 2024-03-28T15:04:39.740 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/36' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/20' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/12' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/39' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/10' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/34' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/38' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/11' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/16' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/32' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/30' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/14' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/18' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/13' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/33' 2024-03-28T15:04:39.741 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/17' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/35' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './188/37' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './188' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/19' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/15' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/31' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/36' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/20' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/12' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/39' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/10' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/34' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/38' 2024-03-28T15:04:39.742 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/11' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/16' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/32' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/30' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/14' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/18' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/13' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/33' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/17' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/35' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './189/37' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './189' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/19' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/15' 2024-03-28T15:04:39.743 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/31' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/36' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/20' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/12' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/39' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/10' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/34' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/38' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/11' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/16' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/32' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/30' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/14' 2024-03-28T15:04:39.744 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/18' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/13' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/33' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/17' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/35' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './190/37' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './190' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/19' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/15' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/31' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/36' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/20' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/12' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/39' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/10' 2024-03-28T15:04:39.745 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/34' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/38' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/11' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/16' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/32' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/30' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/14' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/18' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/13' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/33' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/17' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/35' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './191/37' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './191' 2024-03-28T15:04:39.746 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/19' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/15' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/31' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/36' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/20' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/12' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/39' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/10' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/34' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/38' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/11' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/16' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/32' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/30' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/14' 2024-03-28T15:04:39.747 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/18' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/13' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/33' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/17' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/35' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './192/37' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './192' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/19' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/15' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/31' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/36' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/20' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/12' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/39' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/10' 2024-03-28T15:04:39.748 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/34' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/38' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/11' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/16' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/32' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/30' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/14' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/18' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/13' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/33' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/17' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/35' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './193/37' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './193' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/19' 2024-03-28T15:04:39.749 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/15' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/31' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/36' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/20' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/12' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/39' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/10' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/34' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/38' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/11' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/16' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/32' 2024-03-28T15:04:39.750 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/30' 2024-03-28T15:04:39.751 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/14' 2024-03-28T15:04:39.751 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/18' 2024-03-28T15:04:39.751 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/13' 2024-03-28T15:04:39.751 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/33' 2024-03-28T15:04:39.751 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/17' 2024-03-28T15:04:39.751 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/35' 2024-03-28T15:04:39.751 INFO:tasks.workunit.client.0.smithi186.stdout:removed './194/37' 2024-03-28T15:04:39.752 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './194' 2024-03-28T15:04:39.752 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/19' 2024-03-28T15:04:39.752 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/15' 2024-03-28T15:04:39.752 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/31' 2024-03-28T15:04:39.752 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/36' 2024-03-28T15:04:39.752 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/20' 2024-03-28T15:04:39.752 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/12' 2024-03-28T15:04:39.752 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/39' 2024-03-28T15:04:39.752 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/10' 2024-03-28T15:04:39.752 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/34' 2024-03-28T15:04:39.752 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/38' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/11' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/16' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/32' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/30' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/14' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/18' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/13' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/33' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/17' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/35' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './195/37' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './195' 2024-03-28T15:04:39.753 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/19' 2024-03-28T15:04:39.754 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/15' 2024-03-28T15:04:39.754 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/31' 2024-03-28T15:04:39.754 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/36' 2024-03-28T15:04:39.754 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/20' 2024-03-28T15:04:39.754 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/12' 2024-03-28T15:04:39.754 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/39' 2024-03-28T15:04:39.754 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/10' 2024-03-28T15:04:40.272 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/34' 2024-03-28T15:04:40.272 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/38' 2024-03-28T15:04:40.272 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/11' 2024-03-28T15:04:40.272 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/16' 2024-03-28T15:04:40.272 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/32' 2024-03-28T15:04:40.272 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/30' 2024-03-28T15:04:40.272 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/14' 2024-03-28T15:04:40.272 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/18' 2024-03-28T15:04:40.272 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/13' 2024-03-28T15:04:40.272 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/33' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/17' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/35' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './196/37' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './196' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/19' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/15' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/31' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/36' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/20' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/12' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/39' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/10' 2024-03-28T15:04:40.273 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/34' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/38' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/11' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/16' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/32' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/30' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/14' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/18' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/13' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/33' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/17' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/35' 2024-03-28T15:04:40.274 INFO:tasks.workunit.client.0.smithi186.stdout:removed './197/37' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './197' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/19' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/15' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/31' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/36' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/20' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/12' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/39' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/10' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/34' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/38' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/11' 2024-03-28T15:04:40.275 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/16' 2024-03-28T15:04:40.276 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/32' 2024-03-28T15:04:40.276 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/30' 2024-03-28T15:04:40.276 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/14' 2024-03-28T15:04:40.276 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/18' 2024-03-28T15:04:40.276 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/13' 2024-03-28T15:04:40.276 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/33' 2024-03-28T15:04:40.276 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/17' 2024-03-28T15:04:40.276 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/35' 2024-03-28T15:04:40.276 INFO:tasks.workunit.client.0.smithi186.stdout:removed './198/37' 2024-03-28T15:04:40.276 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './198' 2024-03-28T15:04:40.276 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/19' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/15' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/31' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/36' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/20' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/12' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/39' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/10' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/34' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/38' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/11' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/16' 2024-03-28T15:04:40.277 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/32' 2024-03-28T15:04:40.278 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/30' 2024-03-28T15:04:40.278 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/14' 2024-03-28T15:04:40.278 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/18' 2024-03-28T15:04:40.278 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/13' 2024-03-28T15:04:40.278 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/33' 2024-03-28T15:04:40.278 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/17' 2024-03-28T15:04:40.278 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/35' 2024-03-28T15:04:40.279 INFO:tasks.workunit.client.0.smithi186.stdout:removed './199/37' 2024-03-28T15:04:40.279 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './199' 2024-03-28T15:04:40.279 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/21' 2024-03-28T15:04:40.279 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/27' 2024-03-28T15:04:40.279 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/31' 2024-03-28T15:04:40.279 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/36' 2024-03-28T15:04:40.279 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/20' 2024-03-28T15:04:40.279 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/39' 2024-03-28T15:04:40.279 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/34' 2024-03-28T15:04:40.279 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/38' 2024-03-28T15:04:40.280 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/28' 2024-03-28T15:04:40.280 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/24' 2024-03-28T15:04:40.280 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/29' 2024-03-28T15:04:40.280 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/32' 2024-03-28T15:04:40.280 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/22' 2024-03-28T15:04:40.280 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/25' 2024-03-28T15:04:40.280 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/30' 2024-03-28T15:04:40.280 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/33' 2024-03-28T15:04:40.280 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/23' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/35' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/26' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './200/37' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './200' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/21' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/27' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/31' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/36' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/20' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/39' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/34' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/38' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/28' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/24' 2024-03-28T15:04:40.281 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/29' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/32' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/22' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/25' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/30' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/33' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/23' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/35' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/26' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed './201/37' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './201' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/21' 2024-03-28T15:04:40.282 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/27' 2024-03-28T15:04:40.283 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/31' 2024-03-28T15:04:40.283 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/36' 2024-03-28T15:04:40.283 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/20' 2024-03-28T15:04:40.283 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/39' 2024-03-28T15:04:40.283 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/34' 2024-03-28T15:04:40.283 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/38' 2024-03-28T15:04:40.283 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/28' 2024-03-28T15:04:40.283 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/24' 2024-03-28T15:04:40.284 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/29' 2024-03-28T15:04:40.284 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/32' 2024-03-28T15:04:40.284 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/22' 2024-03-28T15:04:40.284 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/25' 2024-03-28T15:04:40.284 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/30' 2024-03-28T15:04:40.284 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/33' 2024-03-28T15:04:40.284 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/23' 2024-03-28T15:04:40.284 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/35' 2024-03-28T15:04:40.284 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/26' 2024-03-28T15:04:40.284 INFO:tasks.workunit.client.0.smithi186.stdout:removed './202/37' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './202' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/21' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/27' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/31' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/36' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/20' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/39' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/34' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/38' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/28' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/24' 2024-03-28T15:04:40.285 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/29' 2024-03-28T15:04:40.286 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/32' 2024-03-28T15:04:40.286 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/22' 2024-03-28T15:04:40.286 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/25' 2024-03-28T15:04:40.286 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/30' 2024-03-28T15:04:40.286 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/33' 2024-03-28T15:04:40.286 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/23' 2024-03-28T15:04:40.286 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/35' 2024-03-28T15:04:40.286 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/26' 2024-03-28T15:04:40.286 INFO:tasks.workunit.client.0.smithi186.stdout:removed './203/37' 2024-03-28T15:04:40.286 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './203' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/21' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/27' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/31' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/36' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/20' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/39' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/34' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/38' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/28' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/24' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/29' 2024-03-28T15:04:40.287 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/32' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/22' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/25' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/30' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/33' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/23' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/35' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/26' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed './204/37' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './204' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/21' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/27' 2024-03-28T15:04:40.288 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/31' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/36' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/20' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/39' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/34' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/38' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/28' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/24' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/29' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/32' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/22' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/25' 2024-03-28T15:04:40.289 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/30' 2024-03-28T15:04:40.290 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/33' 2024-03-28T15:04:40.290 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/23' 2024-03-28T15:04:40.290 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/35' 2024-03-28T15:04:40.290 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/26' 2024-03-28T15:04:40.290 INFO:tasks.workunit.client.0.smithi186.stdout:removed './205/37' 2024-03-28T15:04:40.290 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './205' 2024-03-28T15:04:40.290 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/21' 2024-03-28T15:04:40.290 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/27' 2024-03-28T15:04:40.290 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/31' 2024-03-28T15:04:40.290 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/36' 2024-03-28T15:04:40.291 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/20' 2024-03-28T15:04:40.291 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/39' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/34' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/38' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/28' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/24' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/29' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/32' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/22' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/25' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/30' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/33' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/23' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/35' 2024-03-28T15:04:40.810 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/26' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './206/37' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './206' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/21' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/27' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/31' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/36' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/20' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/39' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/34' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/38' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/28' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/24' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/29' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/32' 2024-03-28T15:04:40.811 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/22' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/25' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/30' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/33' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/23' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/35' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/26' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './207/37' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './207' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/21' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/27' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/31' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/36' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/20' 2024-03-28T15:04:40.812 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/39' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/34' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/38' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/28' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/24' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/29' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/32' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/22' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/25' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/30' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/33' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/23' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/35' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/26' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed './208/37' 2024-03-28T15:04:40.813 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './208' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/21' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/27' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/31' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/36' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/20' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/39' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/34' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/38' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/28' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/24' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/29' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/32' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/22' 2024-03-28T15:04:40.814 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/25' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/30' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/33' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/23' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/35' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/26' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './209/37' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './209' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/21' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/27' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/31' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/36' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/20' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/39' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/34' 2024-03-28T15:04:40.815 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/38' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/28' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/24' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/29' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/32' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/22' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/25' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/30' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/33' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/23' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/35' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/26' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './210/37' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './210' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/21' 2024-03-28T15:04:40.816 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/27' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/31' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/36' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/20' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/39' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/34' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/38' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/28' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/24' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/29' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/32' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/22' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/25' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/30' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/33' 2024-03-28T15:04:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/23' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/35' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/26' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './211/37' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './211' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/21' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/27' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/31' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/36' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/20' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/39' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/34' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/38' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/28' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/24' 2024-03-28T15:04:40.818 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/29' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/32' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/22' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/25' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/30' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/33' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/23' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/35' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/26' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './212/37' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './212' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/21' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/27' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/31' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/36' 2024-03-28T15:04:40.819 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/20' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/39' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/34' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/38' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/28' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/24' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/29' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/32' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/22' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/25' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/30' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/33' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/23' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/35' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/26' 2024-03-28T15:04:40.820 INFO:tasks.workunit.client.0.smithi186.stdout:removed './213/37' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './213' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/21' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/27' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/31' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/36' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/20' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/39' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/34' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/38' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/28' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/24' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/29' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/32' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/22' 2024-03-28T15:04:40.821 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/25' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/30' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/33' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/23' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/35' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/26' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './214/37' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './214' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/21' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/27' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/31' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/36' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/20' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/39' 2024-03-28T15:04:40.822 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/34' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/38' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/28' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/24' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/29' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/32' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/22' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/25' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/30' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/33' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/23' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/35' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/26' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './215/37' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './215' 2024-03-28T15:04:40.823 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/21' 2024-03-28T15:04:40.824 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/27' 2024-03-28T15:04:40.824 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/31' 2024-03-28T15:04:40.824 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/36' 2024-03-28T15:04:40.824 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/20' 2024-03-28T15:04:40.824 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/39' 2024-03-28T15:04:40.824 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/34' 2024-03-28T15:04:40.824 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/38' 2024-03-28T15:04:41.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/28' 2024-03-28T15:04:41.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/24' 2024-03-28T15:04:41.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/29' 2024-03-28T15:04:41.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/32' 2024-03-28T15:04:41.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/22' 2024-03-28T15:04:41.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/25' 2024-03-28T15:04:41.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/30' 2024-03-28T15:04:41.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/33' 2024-03-28T15:04:41.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/23' 2024-03-28T15:04:41.355 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/35' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/26' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './216/37' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './216' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/21' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/27' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/31' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/36' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/20' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/39' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/34' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/38' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/28' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/24' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/29' 2024-03-28T15:04:41.356 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/32' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/22' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/25' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/30' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/33' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/23' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/35' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/26' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './217/37' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './217' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/21' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/27' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/31' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/36' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/20' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/39' 2024-03-28T15:04:41.357 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/34' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/38' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/28' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/24' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/29' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/32' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/22' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/25' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/30' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/33' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/23' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/35' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/26' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed './218/37' 2024-03-28T15:04:41.358 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './218' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/21' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/27' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/31' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/36' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/20' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/39' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/34' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/38' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/28' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/24' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/29' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/32' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/22' 2024-03-28T15:04:41.359 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/25' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/30' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/33' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/23' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/35' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/26' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './219/37' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './219' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/21' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/27' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/31' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/36' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/20' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/39' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/34' 2024-03-28T15:04:41.360 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/38' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/28' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/24' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/29' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/32' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/22' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/25' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/30' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/33' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/23' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/35' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/26' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './220/37' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './220' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/21' 2024-03-28T15:04:41.361 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/27' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/31' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/36' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/20' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/39' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/34' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/38' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/28' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/24' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/29' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/32' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/22' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/25' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/30' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/33' 2024-03-28T15:04:41.362 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/23' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/35' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/26' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './221/37' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './221' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/21' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/27' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/31' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/36' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/20' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/39' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/34' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/38' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/28' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/24' 2024-03-28T15:04:41.363 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/29' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/32' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/22' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/25' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/30' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/33' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/23' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/35' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/26' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './222/37' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './222' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/21' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/27' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/31' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/36' 2024-03-28T15:04:41.364 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/20' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/39' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/34' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/38' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/28' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/24' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/29' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/32' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/22' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/25' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/30' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/33' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/23' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/35' 2024-03-28T15:04:41.365 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/26' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './223/37' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './223' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/21' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/27' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/31' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/36' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/20' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/39' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/34' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/38' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/28' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/24' 2024-03-28T15:04:41.366 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/29' 2024-03-28T15:04:41.369 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/32' 2024-03-28T15:04:41.369 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/22' 2024-03-28T15:04:41.369 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/25' 2024-03-28T15:04:41.369 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/30' 2024-03-28T15:04:41.369 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/33' 2024-03-28T15:04:41.369 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/23' 2024-03-28T15:04:41.369 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/35' 2024-03-28T15:04:41.369 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/26' 2024-03-28T15:04:41.369 INFO:tasks.workunit.client.0.smithi186.stdout:removed './224/37' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './224' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/21' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/27' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/31' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/36' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/20' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/39' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/34' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/38' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/28' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/24' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/29' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/32' 2024-03-28T15:04:41.370 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/22' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/25' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/30' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/33' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/23' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/35' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/26' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './225/37' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './225' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/21' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/27' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/31' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/36' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/20' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/39' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/34' 2024-03-28T15:04:41.371 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/38' 2024-03-28T15:04:41.372 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/28' 2024-03-28T15:04:41.372 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/24' 2024-03-28T15:04:41.929 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/29' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/32' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/22' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/25' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/30' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/33' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/23' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/35' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/26' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed './226/37' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './226' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/21' 2024-03-28T15:04:41.930 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/27' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/31' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/36' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/20' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/39' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/34' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/38' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/28' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/24' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/29' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/32' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/22' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/25' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/30' 2024-03-28T15:04:41.931 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/33' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/23' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/35' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/26' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './227/37' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './227' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/21' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/27' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/31' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/36' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/20' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/39' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/34' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/38' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/28' 2024-03-28T15:04:41.932 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/24' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/29' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/32' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/22' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/25' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/30' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/33' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/23' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/35' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/26' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './228/37' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './228' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/21' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/27' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/31' 2024-03-28T15:04:41.933 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/36' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/20' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/39' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/34' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/38' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/28' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/24' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/29' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/32' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/22' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/25' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/30' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/33' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/23' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/35' 2024-03-28T15:04:41.934 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/26' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './229/37' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './229' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/21' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/27' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/31' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/36' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/20' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/39' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/34' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/38' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/28' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/24' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/29' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/32' 2024-03-28T15:04:41.935 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/22' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/25' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/30' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/33' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/23' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/35' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/26' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './230/37' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './230' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/21' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/27' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/31' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/36' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/20' 2024-03-28T15:04:41.936 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/39' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/34' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/38' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/28' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/24' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/29' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/32' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/22' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/25' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/30' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/33' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/23' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/35' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/26' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed './231/37' 2024-03-28T15:04:41.937 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './231' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/21' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/27' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/31' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/36' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/20' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/39' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/34' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/38' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/28' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/24' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/29' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/32' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/22' 2024-03-28T15:04:41.938 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/25' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/30' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/33' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/23' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/35' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/26' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './232/37' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './232' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/21' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/27' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/31' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/36' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/20' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/39' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/34' 2024-03-28T15:04:41.939 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/38' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/28' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/24' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/29' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/32' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/22' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/25' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/30' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/33' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/23' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/35' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/26' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './233/37' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './233' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/21' 2024-03-28T15:04:41.940 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/27' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/31' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/36' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/20' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/39' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/34' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/38' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/28' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/24' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/29' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/32' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/22' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/25' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/30' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/33' 2024-03-28T15:04:41.941 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/23' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/35' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/26' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './234/37' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './234' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/21' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/27' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/31' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/36' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/20' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/39' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/34' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/38' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/28' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/24' 2024-03-28T15:04:41.942 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/29' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/32' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/22' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/25' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/30' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/33' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/23' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/35' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/26' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './235/37' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './235' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/21' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/27' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/31' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/36' 2024-03-28T15:04:41.943 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/20' 2024-03-28T15:04:41.944 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/39' 2024-03-28T15:04:41.944 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/34' 2024-03-28T15:04:41.944 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/38' 2024-03-28T15:04:41.944 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/28' 2024-03-28T15:04:41.944 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/24' 2024-03-28T15:04:41.944 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/29' 2024-03-28T15:04:41.944 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/32' 2024-03-28T15:04:42.427 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/22' 2024-03-28T15:04:42.427 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/25' 2024-03-28T15:04:42.427 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/30' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/33' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/23' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/35' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/26' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './236/37' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './236' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/21' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/27' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/31' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/36' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/20' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/39' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/34' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/38' 2024-03-28T15:04:42.428 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/28' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/24' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/29' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/32' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/22' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/25' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/30' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/33' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/23' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/35' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/26' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './237/37' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './237' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/21' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/27' 2024-03-28T15:04:42.429 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/31' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/36' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/20' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/39' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/34' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/38' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/28' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/24' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/29' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/32' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/22' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/25' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/30' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/33' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/23' 2024-03-28T15:04:42.430 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/35' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/26' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './238/37' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './238' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/21' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/27' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/31' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/36' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/20' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/39' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/34' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/38' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/28' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/24' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/29' 2024-03-28T15:04:42.431 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/32' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/22' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/25' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/30' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/33' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/23' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/35' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/26' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './239/37' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './239' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/21' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/27' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/31' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/36' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/20' 2024-03-28T15:04:42.432 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/39' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/34' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/38' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/28' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/24' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/29' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/32' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/22' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/25' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/30' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/33' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/23' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/35' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/26' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './240/37' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './240' 2024-03-28T15:04:42.433 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/21' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/27' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/31' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/36' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/20' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/39' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/34' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/38' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/28' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/24' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/29' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/32' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/22' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/25' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/30' 2024-03-28T15:04:42.434 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/33' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/23' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/35' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/26' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './241/37' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './241' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/21' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/27' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/31' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/36' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/20' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/39' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/34' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/38' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/28' 2024-03-28T15:04:42.435 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/24' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/29' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/32' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/22' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/25' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/30' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/33' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/23' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/35' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/26' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './242/37' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './242' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/21' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/27' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/31' 2024-03-28T15:04:42.436 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/36' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/20' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/39' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/34' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/38' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/28' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/24' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/29' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/32' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/22' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/25' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/30' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/33' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/23' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/35' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/26' 2024-03-28T15:04:42.437 INFO:tasks.workunit.client.0.smithi186.stdout:removed './243/37' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './243' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/21' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/27' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/31' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/36' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/20' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/39' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/34' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/38' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/28' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/24' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/29' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/32' 2024-03-28T15:04:42.438 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/22' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/25' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/30' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/33' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/23' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/35' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/26' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './244/37' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './244' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/21' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/27' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/31' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/36' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/20' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/39' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/34' 2024-03-28T15:04:42.439 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/38' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/28' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/24' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/29' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/32' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/22' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/25' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/30' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/33' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/23' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/35' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/26' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './245/37' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './245' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/21' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/27' 2024-03-28T15:04:42.440 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/31' 2024-03-28T15:04:42.441 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/36' 2024-03-28T15:04:42.441 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/20' 2024-03-28T15:04:42.441 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/39' 2024-03-28T15:04:42.441 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/34' 2024-03-28T15:04:42.441 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/38' 2024-03-28T15:04:42.441 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/28' 2024-03-28T15:04:42.441 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/24' 2024-03-28T15:04:42.441 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/29' 2024-03-28T15:04:42.441 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/32' 2024-03-28T15:04:42.441 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/22' 2024-03-28T15:04:43.010 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/25' 2024-03-28T15:04:43.010 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/30' 2024-03-28T15:04:43.010 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/33' 2024-03-28T15:04:43.010 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/23' 2024-03-28T15:04:43.010 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/35' 2024-03-28T15:04:43.010 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/26' 2024-03-28T15:04:43.010 INFO:tasks.workunit.client.0.smithi186.stdout:removed './246/37' 2024-03-28T15:04:43.010 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './246' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/21' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/27' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/31' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/36' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/20' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/39' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/34' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/38' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/28' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/24' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/29' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/32' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/22' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/25' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/30' 2024-03-28T15:04:43.011 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/33' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/23' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/35' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/26' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './247/37' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './247' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/21' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/27' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/31' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/36' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/20' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/39' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/34' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/38' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/28' 2024-03-28T15:04:43.012 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/24' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/29' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/32' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/22' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/25' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/30' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/33' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/23' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/35' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/26' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './248/37' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './248' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/21' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/27' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/31' 2024-03-28T15:04:43.013 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/36' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/20' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/39' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/34' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/38' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/28' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/24' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/29' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/32' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/22' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/25' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/30' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/33' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/23' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/35' 2024-03-28T15:04:43.014 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/26' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './249/37' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './249' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/21' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/27' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/31' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/36' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/20' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/39' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/34' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/38' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/28' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/24' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/29' 2024-03-28T15:04:43.015 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/32' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/22' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/25' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/30' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/33' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/23' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/35' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/26' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './250/37' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './250' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/21' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/27' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/31' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/36' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/20' 2024-03-28T15:04:43.016 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/39' 2024-03-28T15:04:43.017 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/34' 2024-03-28T15:04:43.017 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/38' 2024-03-28T15:04:43.017 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/28' 2024-03-28T15:04:43.017 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/24' 2024-03-28T15:04:43.017 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/29' 2024-03-28T15:04:43.017 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/32' 2024-03-28T15:04:43.017 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/22' 2024-03-28T15:04:43.017 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/25' 2024-03-28T15:04:43.017 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/30' 2024-03-28T15:04:43.017 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/33' 2024-03-28T15:04:43.017 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/23' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/35' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/26' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './251/37' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './251' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/21' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/27' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/31' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/36' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/20' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/39' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/34' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/38' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/28' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/24' 2024-03-28T15:04:43.018 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/29' 2024-03-28T15:04:43.019 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/32' 2024-03-28T15:04:43.019 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/22' 2024-03-28T15:04:43.019 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/25' 2024-03-28T15:04:43.019 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/30' 2024-03-28T15:04:43.019 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/33' 2024-03-28T15:04:43.019 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/23' 2024-03-28T15:04:43.019 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/35' 2024-03-28T15:04:43.019 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/26' 2024-03-28T15:04:43.019 INFO:tasks.workunit.client.0.smithi186.stdout:removed './252/37' 2024-03-28T15:04:43.019 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './252' 2024-03-28T15:04:43.019 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/21' 2024-03-28T15:04:43.020 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/27' 2024-03-28T15:04:43.020 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/31' 2024-03-28T15:04:43.020 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/36' 2024-03-28T15:04:43.020 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/20' 2024-03-28T15:04:43.020 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/39' 2024-03-28T15:04:43.020 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/34' 2024-03-28T15:04:43.020 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/38' 2024-03-28T15:04:43.020 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/28' 2024-03-28T15:04:43.020 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/24' 2024-03-28T15:04:43.020 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/29' 2024-03-28T15:04:43.020 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/32' 2024-03-28T15:04:43.021 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/22' 2024-03-28T15:04:43.021 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/25' 2024-03-28T15:04:43.021 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/30' 2024-03-28T15:04:43.021 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/33' 2024-03-28T15:04:43.021 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/23' 2024-03-28T15:04:43.021 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/35' 2024-03-28T15:04:43.021 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/26' 2024-03-28T15:04:43.021 INFO:tasks.workunit.client.0.smithi186.stdout:removed './253/37' 2024-03-28T15:04:43.021 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './253' 2024-03-28T15:04:43.021 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/21' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/27' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/31' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/36' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/20' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/39' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/34' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/38' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/28' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/24' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/29' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/32' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/22' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/25' 2024-03-28T15:04:43.022 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/30' 2024-03-28T15:04:43.023 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/33' 2024-03-28T15:04:43.023 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/23' 2024-03-28T15:04:43.023 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/35' 2024-03-28T15:04:43.023 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/26' 2024-03-28T15:04:43.023 INFO:tasks.workunit.client.0.smithi186.stdout:removed './254/37' 2024-03-28T15:04:43.023 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './254' 2024-03-28T15:04:43.023 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/21' 2024-03-28T15:04:43.023 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/27' 2024-03-28T15:04:43.023 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/31' 2024-03-28T15:04:43.023 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/36' 2024-03-28T15:04:43.023 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/20' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/39' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/34' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/38' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/28' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/24' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/29' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/32' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/22' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/25' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/30' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/33' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/23' 2024-03-28T15:04:43.024 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/35' 2024-03-28T15:04:43.025 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/26' 2024-03-28T15:04:43.025 INFO:tasks.workunit.client.0.smithi186.stdout:removed './255/37' 2024-03-28T15:04:43.025 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './255' 2024-03-28T15:04:43.025 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/21' 2024-03-28T15:04:43.025 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/27' 2024-03-28T15:04:43.025 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/31' 2024-03-28T15:04:43.025 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/36' 2024-03-28T15:04:43.025 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/20' 2024-03-28T15:04:43.025 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/39' 2024-03-28T15:04:43.025 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/34' 2024-03-28T15:04:43.025 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/38' 2024-03-28T15:04:43.026 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/28' 2024-03-28T15:04:43.026 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/24' 2024-03-28T15:04:43.026 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/29' 2024-03-28T15:04:43.026 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/32' 2024-03-28T15:04:43.026 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/22' 2024-03-28T15:04:43.026 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/25' 2024-03-28T15:04:43.026 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/30' 2024-03-28T15:04:43.566 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/33' 2024-03-28T15:04:43.567 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/23' 2024-03-28T15:04:43.567 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/35' 2024-03-28T15:04:43.567 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/26' 2024-03-28T15:04:43.567 INFO:tasks.workunit.client.0.smithi186.stdout:removed './256/37' 2024-03-28T15:04:43.567 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './256' 2024-03-28T15:04:43.567 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/21' 2024-03-28T15:04:43.568 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/27' 2024-03-28T15:04:43.568 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/31' 2024-03-28T15:04:43.568 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/36' 2024-03-28T15:04:43.568 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/20' 2024-03-28T15:04:43.568 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/39' 2024-03-28T15:04:43.568 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/34' 2024-03-28T15:04:43.568 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/38' 2024-03-28T15:04:43.569 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/28' 2024-03-28T15:04:43.569 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/24' 2024-03-28T15:04:43.569 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/29' 2024-03-28T15:04:43.569 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/32' 2024-03-28T15:04:43.569 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/22' 2024-03-28T15:04:43.569 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/25' 2024-03-28T15:04:43.569 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/30' 2024-03-28T15:04:43.570 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/33' 2024-03-28T15:04:43.570 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/23' 2024-03-28T15:04:43.570 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/35' 2024-03-28T15:04:43.570 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/26' 2024-03-28T15:04:43.570 INFO:tasks.workunit.client.0.smithi186.stdout:removed './257/37' 2024-03-28T15:04:43.570 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './257' 2024-03-28T15:04:43.571 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/21' 2024-03-28T15:04:43.571 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/27' 2024-03-28T15:04:43.571 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/31' 2024-03-28T15:04:43.571 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/36' 2024-03-28T15:04:43.571 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/20' 2024-03-28T15:04:43.571 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/39' 2024-03-28T15:04:43.572 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/34' 2024-03-28T15:04:43.572 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/38' 2024-03-28T15:04:43.572 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/28' 2024-03-28T15:04:43.572 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/24' 2024-03-28T15:04:43.572 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/29' 2024-03-28T15:04:43.572 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/32' 2024-03-28T15:04:43.572 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/22' 2024-03-28T15:04:43.573 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/25' 2024-03-28T15:04:43.573 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/30' 2024-03-28T15:04:43.573 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/33' 2024-03-28T15:04:43.573 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/23' 2024-03-28T15:04:43.574 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/35' 2024-03-28T15:04:43.574 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/26' 2024-03-28T15:04:43.574 INFO:tasks.workunit.client.0.smithi186.stdout:removed './258/37' 2024-03-28T15:04:43.574 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './258' 2024-03-28T15:04:43.574 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/21' 2024-03-28T15:04:43.574 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/27' 2024-03-28T15:04:43.575 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/31' 2024-03-28T15:04:43.575 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/36' 2024-03-28T15:04:43.575 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/20' 2024-03-28T15:04:43.575 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/39' 2024-03-28T15:04:43.575 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/34' 2024-03-28T15:04:43.575 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/38' 2024-03-28T15:04:43.576 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/28' 2024-03-28T15:04:43.576 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/24' 2024-03-28T15:04:43.576 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/29' 2024-03-28T15:04:43.576 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/32' 2024-03-28T15:04:43.576 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/22' 2024-03-28T15:04:43.576 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/25' 2024-03-28T15:04:43.577 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/30' 2024-03-28T15:04:43.577 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/33' 2024-03-28T15:04:43.577 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/23' 2024-03-28T15:04:43.577 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/35' 2024-03-28T15:04:43.577 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/26' 2024-03-28T15:04:43.577 INFO:tasks.workunit.client.0.smithi186.stdout:removed './259/37' 2024-03-28T15:04:43.578 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './259' 2024-03-28T15:04:43.578 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/21' 2024-03-28T15:04:43.578 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/27' 2024-03-28T15:04:43.578 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/31' 2024-03-28T15:04:43.578 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/36' 2024-03-28T15:04:43.578 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/20' 2024-03-28T15:04:43.579 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/39' 2024-03-28T15:04:43.579 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/34' 2024-03-28T15:04:43.579 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/38' 2024-03-28T15:04:43.579 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/28' 2024-03-28T15:04:43.579 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/24' 2024-03-28T15:04:43.579 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/29' 2024-03-28T15:04:43.579 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/32' 2024-03-28T15:04:43.580 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/22' 2024-03-28T15:04:43.580 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/25' 2024-03-28T15:04:43.580 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/30' 2024-03-28T15:04:43.580 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/33' 2024-03-28T15:04:43.580 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/23' 2024-03-28T15:04:43.580 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/35' 2024-03-28T15:04:43.581 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/26' 2024-03-28T15:04:43.581 INFO:tasks.workunit.client.0.smithi186.stdout:removed './260/37' 2024-03-28T15:04:43.581 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './260' 2024-03-28T15:04:43.581 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/21' 2024-03-28T15:04:43.581 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/27' 2024-03-28T15:04:43.581 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/31' 2024-03-28T15:04:43.582 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/36' 2024-03-28T15:04:43.582 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/20' 2024-03-28T15:04:43.582 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/39' 2024-03-28T15:04:43.582 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/34' 2024-03-28T15:04:43.582 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/38' 2024-03-28T15:04:43.582 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/28' 2024-03-28T15:04:43.583 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/24' 2024-03-28T15:04:43.583 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/29' 2024-03-28T15:04:43.583 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/32' 2024-03-28T15:04:43.583 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/22' 2024-03-28T15:04:43.583 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/25' 2024-03-28T15:04:43.583 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/30' 2024-03-28T15:04:43.584 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/33' 2024-03-28T15:04:43.584 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/23' 2024-03-28T15:04:43.584 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/35' 2024-03-28T15:04:43.584 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/26' 2024-03-28T15:04:43.584 INFO:tasks.workunit.client.0.smithi186.stdout:removed './261/37' 2024-03-28T15:04:43.584 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './261' 2024-03-28T15:04:43.584 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/21' 2024-03-28T15:04:43.585 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/27' 2024-03-28T15:04:43.585 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/31' 2024-03-28T15:04:43.585 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/36' 2024-03-28T15:04:43.585 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/20' 2024-03-28T15:04:43.585 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/39' 2024-03-28T15:04:43.585 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/34' 2024-03-28T15:04:43.586 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/38' 2024-03-28T15:04:43.586 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/28' 2024-03-28T15:04:43.586 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/24' 2024-03-28T15:04:43.586 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/29' 2024-03-28T15:04:43.586 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/32' 2024-03-28T15:04:43.586 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/22' 2024-03-28T15:04:43.587 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/25' 2024-03-28T15:04:43.587 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/30' 2024-03-28T15:04:43.587 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/33' 2024-03-28T15:04:43.587 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/23' 2024-03-28T15:04:43.587 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/35' 2024-03-28T15:04:43.587 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/26' 2024-03-28T15:04:43.588 INFO:tasks.workunit.client.0.smithi186.stdout:removed './262/37' 2024-03-28T15:04:43.588 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './262' 2024-03-28T15:04:43.588 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/21' 2024-03-28T15:04:43.588 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/27' 2024-03-28T15:04:43.588 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/31' 2024-03-28T15:04:43.588 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/36' 2024-03-28T15:04:43.589 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/20' 2024-03-28T15:04:43.589 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/39' 2024-03-28T15:04:43.589 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/34' 2024-03-28T15:04:43.589 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/38' 2024-03-28T15:04:43.589 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/28' 2024-03-28T15:04:43.589 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/24' 2024-03-28T15:04:43.589 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/29' 2024-03-28T15:04:43.590 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/32' 2024-03-28T15:04:43.590 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/22' 2024-03-28T15:04:43.590 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/25' 2024-03-28T15:04:43.590 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/30' 2024-03-28T15:04:43.590 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/33' 2024-03-28T15:04:43.590 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/23' 2024-03-28T15:04:43.591 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/35' 2024-03-28T15:04:43.591 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/26' 2024-03-28T15:04:43.591 INFO:tasks.workunit.client.0.smithi186.stdout:removed './263/37' 2024-03-28T15:04:43.591 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './263' 2024-03-28T15:04:43.591 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/21' 2024-03-28T15:04:43.591 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/27' 2024-03-28T15:04:43.592 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/31' 2024-03-28T15:04:43.592 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/36' 2024-03-28T15:04:43.592 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/20' 2024-03-28T15:04:43.592 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/39' 2024-03-28T15:04:43.592 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/34' 2024-03-28T15:04:43.592 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/38' 2024-03-28T15:04:43.592 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/28' 2024-03-28T15:04:43.593 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/24' 2024-03-28T15:04:43.593 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/29' 2024-03-28T15:04:43.593 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/32' 2024-03-28T15:04:43.593 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/22' 2024-03-28T15:04:43.593 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/25' 2024-03-28T15:04:43.593 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/30' 2024-03-28T15:04:43.593 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/33' 2024-03-28T15:04:43.593 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/23' 2024-03-28T15:04:43.593 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/35' 2024-03-28T15:04:43.594 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/26' 2024-03-28T15:04:43.594 INFO:tasks.workunit.client.0.smithi186.stdout:removed './264/37' 2024-03-28T15:04:43.594 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './264' 2024-03-28T15:04:43.594 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/21' 2024-03-28T15:04:43.594 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/27' 2024-03-28T15:04:43.594 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/31' 2024-03-28T15:04:43.594 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/36' 2024-03-28T15:04:43.594 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/20' 2024-03-28T15:04:43.595 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/39' 2024-03-28T15:04:43.595 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/34' 2024-03-28T15:04:43.595 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/38' 2024-03-28T15:04:43.595 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/28' 2024-03-28T15:04:43.595 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/24' 2024-03-28T15:04:43.595 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/29' 2024-03-28T15:04:43.595 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/32' 2024-03-28T15:04:43.595 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/22' 2024-03-28T15:04:43.595 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/25' 2024-03-28T15:04:43.596 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/30' 2024-03-28T15:04:43.596 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/33' 2024-03-28T15:04:43.596 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/23' 2024-03-28T15:04:43.596 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/35' 2024-03-28T15:04:43.596 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/26' 2024-03-28T15:04:43.596 INFO:tasks.workunit.client.0.smithi186.stdout:removed './265/37' 2024-03-28T15:04:43.596 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './265' 2024-03-28T15:04:43.597 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/21' 2024-03-28T15:04:43.597 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/27' 2024-03-28T15:04:43.597 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/31' 2024-03-28T15:04:43.597 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/36' 2024-03-28T15:04:43.597 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/20' 2024-03-28T15:04:43.597 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/39' 2024-03-28T15:04:43.597 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/34' 2024-03-28T15:04:43.597 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/38' 2024-03-28T15:04:43.598 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/28' 2024-03-28T15:04:43.598 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/24' 2024-03-28T15:04:43.598 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/29' 2024-03-28T15:04:43.598 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/32' 2024-03-28T15:04:43.598 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/22' 2024-03-28T15:04:43.598 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/25' 2024-03-28T15:04:43.598 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/30' 2024-03-28T15:04:43.598 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/33' 2024-03-28T15:04:43.598 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/23' 2024-03-28T15:04:44.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/35' 2024-03-28T15:04:44.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/26' 2024-03-28T15:04:44.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './266/37' 2024-03-28T15:04:44.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './266' 2024-03-28T15:04:44.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/21' 2024-03-28T15:04:44.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/27' 2024-03-28T15:04:44.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/31' 2024-03-28T15:04:44.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/36' 2024-03-28T15:04:44.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/20' 2024-03-28T15:04:44.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/39' 2024-03-28T15:04:44.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/34' 2024-03-28T15:04:44.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/38' 2024-03-28T15:04:44.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/28' 2024-03-28T15:04:44.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/24' 2024-03-28T15:04:44.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/29' 2024-03-28T15:04:44.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/32' 2024-03-28T15:04:44.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/22' 2024-03-28T15:04:44.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/25' 2024-03-28T15:04:44.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/30' 2024-03-28T15:04:44.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/33' 2024-03-28T15:04:44.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/23' 2024-03-28T15:04:44.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/35' 2024-03-28T15:04:44.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/26' 2024-03-28T15:04:44.134 INFO:tasks.workunit.client.0.smithi186.stdout:removed './267/37' 2024-03-28T15:04:44.134 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './267' 2024-03-28T15:04:44.134 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/21' 2024-03-28T15:04:44.134 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/27' 2024-03-28T15:04:44.135 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/31' 2024-03-28T15:04:44.135 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/36' 2024-03-28T15:04:44.135 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/20' 2024-03-28T15:04:44.135 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/39' 2024-03-28T15:04:44.135 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/34' 2024-03-28T15:04:44.135 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/38' 2024-03-28T15:04:44.136 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/28' 2024-03-28T15:04:44.136 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/24' 2024-03-28T15:04:44.136 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/29' 2024-03-28T15:04:44.136 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/32' 2024-03-28T15:04:44.136 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/22' 2024-03-28T15:04:44.136 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/25' 2024-03-28T15:04:44.136 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/30' 2024-03-28T15:04:44.137 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/33' 2024-03-28T15:04:44.137 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/23' 2024-03-28T15:04:44.137 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/35' 2024-03-28T15:04:44.137 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/26' 2024-03-28T15:04:44.137 INFO:tasks.workunit.client.0.smithi186.stdout:removed './268/37' 2024-03-28T15:04:44.137 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './268' 2024-03-28T15:04:44.137 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/21' 2024-03-28T15:04:44.138 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/27' 2024-03-28T15:04:44.138 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/31' 2024-03-28T15:04:44.138 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/36' 2024-03-28T15:04:44.138 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/20' 2024-03-28T15:04:44.138 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/39' 2024-03-28T15:04:44.138 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/34' 2024-03-28T15:04:44.138 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/38' 2024-03-28T15:04:44.138 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/28' 2024-03-28T15:04:44.139 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/24' 2024-03-28T15:04:44.139 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/29' 2024-03-28T15:04:44.139 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/32' 2024-03-28T15:04:44.139 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/22' 2024-03-28T15:04:44.139 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/25' 2024-03-28T15:04:44.139 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/30' 2024-03-28T15:04:44.139 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/33' 2024-03-28T15:04:44.139 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/23' 2024-03-28T15:04:44.139 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/35' 2024-03-28T15:04:44.139 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/26' 2024-03-28T15:04:44.140 INFO:tasks.workunit.client.0.smithi186.stdout:removed './269/37' 2024-03-28T15:04:44.140 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './269' 2024-03-28T15:04:44.140 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/21' 2024-03-28T15:04:44.140 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/27' 2024-03-28T15:04:44.140 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/31' 2024-03-28T15:04:44.140 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/36' 2024-03-28T15:04:44.140 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/20' 2024-03-28T15:04:44.140 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/39' 2024-03-28T15:04:44.140 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/34' 2024-03-28T15:04:44.141 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/38' 2024-03-28T15:04:44.141 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/28' 2024-03-28T15:04:44.141 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/24' 2024-03-28T15:04:44.141 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/29' 2024-03-28T15:04:44.141 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/32' 2024-03-28T15:04:44.141 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/22' 2024-03-28T15:04:44.141 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/25' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/30' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/33' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/23' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/35' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/26' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed './270/37' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './270' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/21' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/27' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/31' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/36' 2024-03-28T15:04:44.142 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/20' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/39' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/34' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/38' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/28' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/24' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/29' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/32' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/22' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/25' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/30' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/33' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/23' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/35' 2024-03-28T15:04:44.143 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/26' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './271/37' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './271' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/21' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/27' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/31' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/36' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/20' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/39' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/34' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/38' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/28' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/24' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/29' 2024-03-28T15:04:44.144 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/32' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/22' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/25' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/30' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/33' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/23' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/35' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/26' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './272/37' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './272' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/21' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/27' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/31' 2024-03-28T15:04:44.145 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/36' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/20' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/39' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/34' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/38' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/28' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/24' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/29' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/32' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/22' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/25' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/30' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/33' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/23' 2024-03-28T15:04:44.146 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/35' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/26' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './273/37' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './273' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/21' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/27' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/31' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/36' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/20' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/39' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/34' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/38' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/28' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/24' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/29' 2024-03-28T15:04:44.147 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/32' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/22' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/25' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/30' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/33' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/23' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/35' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/26' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './274/37' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './274' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/21' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/27' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/31' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/36' 2024-03-28T15:04:44.148 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/20' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/39' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/34' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/38' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/28' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/24' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/29' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/32' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/22' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/25' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/30' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/33' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/23' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/35' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/26' 2024-03-28T15:04:44.149 INFO:tasks.workunit.client.0.smithi186.stdout:removed './275/37' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './275' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/21' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/27' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/31' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/36' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/20' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/39' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/34' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/38' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/28' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/24' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/29' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/32' 2024-03-28T15:04:44.150 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/22' 2024-03-28T15:04:44.151 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/25' 2024-03-28T15:04:44.151 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/30' 2024-03-28T15:04:44.151 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/33' 2024-03-28T15:04:44.151 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/23' 2024-03-28T15:04:44.151 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/35' 2024-03-28T15:04:44.151 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/26' 2024-03-28T15:04:44.630 INFO:tasks.workunit.client.0.smithi186.stdout:removed './276/37' 2024-03-28T15:04:44.631 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './276' 2024-03-28T15:04:44.631 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/21' 2024-03-28T15:04:44.631 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/27' 2024-03-28T15:04:44.631 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/31' 2024-03-28T15:04:44.633 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/36' 2024-03-28T15:04:44.633 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/20' 2024-03-28T15:04:44.633 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/39' 2024-03-28T15:04:44.634 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/34' 2024-03-28T15:04:44.634 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/38' 2024-03-28T15:04:44.634 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/28' 2024-03-28T15:04:44.634 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/24' 2024-03-28T15:04:44.634 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/29' 2024-03-28T15:04:44.634 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/32' 2024-03-28T15:04:44.634 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/22' 2024-03-28T15:04:44.634 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/25' 2024-03-28T15:04:44.635 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/30' 2024-03-28T15:04:44.635 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/33' 2024-03-28T15:04:44.635 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/23' 2024-03-28T15:04:44.635 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/35' 2024-03-28T15:04:44.635 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/26' 2024-03-28T15:04:44.635 INFO:tasks.workunit.client.0.smithi186.stdout:removed './277/37' 2024-03-28T15:04:44.635 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './277' 2024-03-28T15:04:44.635 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/21' 2024-03-28T15:04:44.636 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/27' 2024-03-28T15:04:44.636 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/31' 2024-03-28T15:04:44.636 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/36' 2024-03-28T15:04:44.636 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/20' 2024-03-28T15:04:44.636 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/39' 2024-03-28T15:04:44.636 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/34' 2024-03-28T15:04:44.636 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/38' 2024-03-28T15:04:44.637 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/28' 2024-03-28T15:04:44.637 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/24' 2024-03-28T15:04:44.637 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/29' 2024-03-28T15:04:44.637 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/32' 2024-03-28T15:04:44.637 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/22' 2024-03-28T15:04:44.637 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/25' 2024-03-28T15:04:44.637 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/30' 2024-03-28T15:04:44.637 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/33' 2024-03-28T15:04:44.638 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/23' 2024-03-28T15:04:44.638 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/35' 2024-03-28T15:04:44.638 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/26' 2024-03-28T15:04:44.638 INFO:tasks.workunit.client.0.smithi186.stdout:removed './278/37' 2024-03-28T15:04:44.638 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './278' 2024-03-28T15:04:44.638 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/21' 2024-03-28T15:04:44.638 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/27' 2024-03-28T15:04:44.639 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/31' 2024-03-28T15:04:44.639 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/36' 2024-03-28T15:04:44.639 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/20' 2024-03-28T15:04:44.639 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/39' 2024-03-28T15:04:44.639 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/34' 2024-03-28T15:04:44.639 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/38' 2024-03-28T15:04:44.639 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/28' 2024-03-28T15:04:44.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/24' 2024-03-28T15:04:44.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/29' 2024-03-28T15:04:44.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/32' 2024-03-28T15:04:44.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/22' 2024-03-28T15:04:44.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/25' 2024-03-28T15:04:44.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/30' 2024-03-28T15:04:44.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/33' 2024-03-28T15:04:44.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/23' 2024-03-28T15:04:44.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/35' 2024-03-28T15:04:44.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/26' 2024-03-28T15:04:44.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './279/37' 2024-03-28T15:04:44.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './279' 2024-03-28T15:04:44.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/21' 2024-03-28T15:04:44.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/27' 2024-03-28T15:04:44.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/31' 2024-03-28T15:04:44.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/36' 2024-03-28T15:04:44.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/20' 2024-03-28T15:04:44.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/39' 2024-03-28T15:04:44.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/34' 2024-03-28T15:04:44.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/38' 2024-03-28T15:04:44.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/28' 2024-03-28T15:04:44.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/24' 2024-03-28T15:04:44.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/29' 2024-03-28T15:04:44.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/32' 2024-03-28T15:04:44.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/22' 2024-03-28T15:04:44.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/25' 2024-03-28T15:04:44.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/30' 2024-03-28T15:04:44.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/33' 2024-03-28T15:04:44.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/23' 2024-03-28T15:04:44.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/35' 2024-03-28T15:04:44.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/26' 2024-03-28T15:04:44.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './280/37' 2024-03-28T15:04:44.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './280' 2024-03-28T15:04:44.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/21' 2024-03-28T15:04:44.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/27' 2024-03-28T15:04:44.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/31' 2024-03-28T15:04:44.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/36' 2024-03-28T15:04:44.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/20' 2024-03-28T15:04:44.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/39' 2024-03-28T15:04:44.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/34' 2024-03-28T15:04:44.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/38' 2024-03-28T15:04:44.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/28' 2024-03-28T15:04:44.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/24' 2024-03-28T15:04:44.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/29' 2024-03-28T15:04:44.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/32' 2024-03-28T15:04:44.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/22' 2024-03-28T15:04:44.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/25' 2024-03-28T15:04:44.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/30' 2024-03-28T15:04:44.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/33' 2024-03-28T15:04:44.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/23' 2024-03-28T15:04:44.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/35' 2024-03-28T15:04:44.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/26' 2024-03-28T15:04:44.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './281/37' 2024-03-28T15:04:44.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './281' 2024-03-28T15:04:44.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/21' 2024-03-28T15:04:44.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/27' 2024-03-28T15:04:44.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/31' 2024-03-28T15:04:44.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/36' 2024-03-28T15:04:44.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/20' 2024-03-28T15:04:44.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/39' 2024-03-28T15:04:44.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/34' 2024-03-28T15:04:44.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/38' 2024-03-28T15:04:44.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/28' 2024-03-28T15:04:44.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/24' 2024-03-28T15:04:44.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/29' 2024-03-28T15:04:44.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/32' 2024-03-28T15:04:44.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/22' 2024-03-28T15:04:44.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/25' 2024-03-28T15:04:44.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/30' 2024-03-28T15:04:44.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/33' 2024-03-28T15:04:44.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/23' 2024-03-28T15:04:44.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/35' 2024-03-28T15:04:44.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/26' 2024-03-28T15:04:44.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './282/37' 2024-03-28T15:04:44.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './282' 2024-03-28T15:04:44.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/21' 2024-03-28T15:04:44.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/27' 2024-03-28T15:04:44.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/31' 2024-03-28T15:04:44.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/36' 2024-03-28T15:04:44.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/20' 2024-03-28T15:04:44.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/39' 2024-03-28T15:04:44.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/34' 2024-03-28T15:04:44.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/38' 2024-03-28T15:04:44.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/28' 2024-03-28T15:04:44.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/24' 2024-03-28T15:04:44.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/29' 2024-03-28T15:04:44.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/32' 2024-03-28T15:04:44.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/22' 2024-03-28T15:04:44.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/25' 2024-03-28T15:04:44.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/30' 2024-03-28T15:04:44.652 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/33' 2024-03-28T15:04:44.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/23' 2024-03-28T15:04:44.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/35' 2024-03-28T15:04:44.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/26' 2024-03-28T15:04:44.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './283/37' 2024-03-28T15:04:44.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './283' 2024-03-28T15:04:44.653 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/21' 2024-03-28T15:04:44.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/27' 2024-03-28T15:04:44.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/31' 2024-03-28T15:04:44.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/36' 2024-03-28T15:04:44.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/20' 2024-03-28T15:04:44.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/39' 2024-03-28T15:04:44.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/34' 2024-03-28T15:04:44.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/38' 2024-03-28T15:04:44.654 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/28' 2024-03-28T15:04:44.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/24' 2024-03-28T15:04:44.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/29' 2024-03-28T15:04:44.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/32' 2024-03-28T15:04:44.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/22' 2024-03-28T15:04:44.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/25' 2024-03-28T15:04:44.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/30' 2024-03-28T15:04:44.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/33' 2024-03-28T15:04:44.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/23' 2024-03-28T15:04:44.655 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/35' 2024-03-28T15:04:44.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/26' 2024-03-28T15:04:44.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './284/37' 2024-03-28T15:04:44.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './284' 2024-03-28T15:04:44.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/21' 2024-03-28T15:04:44.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/27' 2024-03-28T15:04:44.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/31' 2024-03-28T15:04:44.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/36' 2024-03-28T15:04:44.656 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/20' 2024-03-28T15:04:44.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/39' 2024-03-28T15:04:44.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/34' 2024-03-28T15:04:44.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/38' 2024-03-28T15:04:44.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/28' 2024-03-28T15:04:44.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/24' 2024-03-28T15:04:44.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/29' 2024-03-28T15:04:44.657 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/32' 2024-03-28T15:04:44.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/22' 2024-03-28T15:04:44.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/25' 2024-03-28T15:04:44.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/30' 2024-03-28T15:04:44.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/33' 2024-03-28T15:04:44.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/23' 2024-03-28T15:04:44.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/35' 2024-03-28T15:04:44.658 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/26' 2024-03-28T15:04:44.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './285/37' 2024-03-28T15:04:44.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './285' 2024-03-28T15:04:44.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/21' 2024-03-28T15:04:44.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/27' 2024-03-28T15:04:44.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/31' 2024-03-28T15:04:44.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/36' 2024-03-28T15:04:44.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/20' 2024-03-28T15:04:44.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/39' 2024-03-28T15:04:44.659 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/34' 2024-03-28T15:04:44.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/38' 2024-03-28T15:04:44.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/28' 2024-03-28T15:04:44.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/24' 2024-03-28T15:04:44.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/29' 2024-03-28T15:04:44.660 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/32' 2024-03-28T15:04:44.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/22' 2024-03-28T15:04:44.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/25' 2024-03-28T15:04:44.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/30' 2024-03-28T15:04:44.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/33' 2024-03-28T15:04:44.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/23' 2024-03-28T15:04:44.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/35' 2024-03-28T15:04:44.661 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/26' 2024-03-28T15:04:44.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed './286/37' 2024-03-28T15:04:44.662 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './286' 2024-03-28T15:04:45.186 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/21' 2024-03-28T15:04:45.186 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/27' 2024-03-28T15:04:45.186 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/31' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/36' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/20' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/39' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/34' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/38' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/28' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/24' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/29' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/32' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/22' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/25' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/30' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/33' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/23' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/35' 2024-03-28T15:04:45.187 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/26' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './287/37' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './287' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/21' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/27' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/31' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/36' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/20' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/39' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/34' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/38' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/28' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/24' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/29' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/32' 2024-03-28T15:04:45.188 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/22' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/25' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/30' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/33' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/23' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/35' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/26' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './288/37' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './288' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/21' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/27' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/31' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/36' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/20' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/39' 2024-03-28T15:04:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/34' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/38' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/28' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/24' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/29' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/32' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/22' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/25' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/30' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/33' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/23' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/35' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/26' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './289/37' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './289' 2024-03-28T15:04:45.190 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/21' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/27' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/31' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/36' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/20' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/39' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/34' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/38' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/28' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/24' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/29' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/32' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/22' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/25' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/30' 2024-03-28T15:04:45.191 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/33' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/23' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/35' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/26' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './290/37' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './290' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/21' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/27' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/31' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/36' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/20' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/39' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/34' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/38' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/28' 2024-03-28T15:04:45.192 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/24' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/29' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/32' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/22' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/25' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/30' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/33' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/23' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/35' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/26' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './291/37' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './291' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/21' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/27' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/31' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/36' 2024-03-28T15:04:45.193 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/20' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/39' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/34' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/38' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/28' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/24' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/29' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/32' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/22' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/25' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/30' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/33' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/23' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/35' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/26' 2024-03-28T15:04:45.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './292/37' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './292' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/21' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/27' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/31' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/36' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/20' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/39' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/34' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/38' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/28' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/24' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/29' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/32' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/22' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/25' 2024-03-28T15:04:45.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/30' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/33' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/23' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/35' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/26' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './293/37' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './293' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/21' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/27' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/31' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/36' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/20' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/39' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/34' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/38' 2024-03-28T15:04:45.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/28' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/24' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/29' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/32' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/22' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/25' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/30' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/33' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/23' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/35' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/26' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './294/37' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './294' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/21' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/27' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/31' 2024-03-28T15:04:45.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/36' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/20' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/39' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/34' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/38' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/28' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/24' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/29' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/32' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/22' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/25' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/30' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/33' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/23' 2024-03-28T15:04:45.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/35' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/26' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './295/37' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './295' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/21' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/27' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/31' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/36' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/20' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/39' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/34' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/38' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/28' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/24' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/29' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/32' 2024-03-28T15:04:45.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/22' 2024-03-28T15:04:45.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/25' 2024-03-28T15:04:45.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/30' 2024-03-28T15:04:45.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/33' 2024-03-28T15:04:45.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/23' 2024-03-28T15:04:45.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/35' 2024-03-28T15:04:45.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/26' 2024-03-28T15:04:45.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './296/37' 2024-03-28T15:04:45.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './296' 2024-03-28T15:04:45.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/21' 2024-03-28T15:04:45.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/27' 2024-03-28T15:04:45.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/31' 2024-03-28T15:04:45.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/36' 2024-03-28T15:04:45.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/20' 2024-03-28T15:04:45.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/39' 2024-03-28T15:04:45.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/34' 2024-03-28T15:04:45.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/38' 2024-03-28T15:04:45.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/28' 2024-03-28T15:04:45.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/24' 2024-03-28T15:04:45.640 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/29' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/32' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/22' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/25' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/30' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/33' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/23' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/35' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/26' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './297/37' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './297' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/21' 2024-03-28T15:04:45.641 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/27' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/31' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/36' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/20' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/39' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/34' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/38' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/28' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/24' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/29' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/32' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/22' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/25' 2024-03-28T15:04:45.642 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/30' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/33' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/23' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/35' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/26' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './298/37' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './298' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/21' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/27' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/31' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/36' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/20' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/39' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/34' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/38' 2024-03-28T15:04:45.643 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/28' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/24' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/29' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/32' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/22' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/25' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/30' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/33' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/23' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/35' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/26' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './299/37' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './299' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/166/19' 2024-03-28T15:04:45.644 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/166/15' 2024-03-28T15:04:45.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/166/20' 2024-03-28T15:04:45.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/166/12' 2024-03-28T15:04:45.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/166/10' 2024-03-28T15:04:45.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/166/11' 2024-03-28T15:04:45.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/166/16' 2024-03-28T15:04:45.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/166/14' 2024-03-28T15:04:45.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/166/18' 2024-03-28T15:04:45.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/166/13' 2024-03-28T15:04:45.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/166/17' 2024-03-28T15:04:45.645 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/166' 2024-03-28T15:04:45.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/147/19' 2024-03-28T15:04:45.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/147/15' 2024-03-28T15:04:45.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/147/20' 2024-03-28T15:04:45.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/147/12' 2024-03-28T15:04:45.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/147/10' 2024-03-28T15:04:45.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/147/11' 2024-03-28T15:04:45.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/147/16' 2024-03-28T15:04:45.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/147/14' 2024-03-28T15:04:45.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/147/18' 2024-03-28T15:04:45.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/147/13' 2024-03-28T15:04:45.646 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/147/17' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/147' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/151/19' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/151/15' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/151/20' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/151/12' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/151/10' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/151/11' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/151/16' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/151/14' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/151/18' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/151/13' 2024-03-28T15:04:45.647 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/151/17' 2024-03-28T15:04:45.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/151' 2024-03-28T15:04:45.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/170/19' 2024-03-28T15:04:45.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/170/15' 2024-03-28T15:04:45.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/170/20' 2024-03-28T15:04:45.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/170/12' 2024-03-28T15:04:45.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/170/10' 2024-03-28T15:04:45.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/170/11' 2024-03-28T15:04:45.648 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/170/16' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/170/14' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/170/18' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/170/13' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/170/17' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/170' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/167/19' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/167/15' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/167/20' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/167/12' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/167/10' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/167/11' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/167/16' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/167/14' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/167/18' 2024-03-28T15:04:45.649 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/167/13' 2024-03-28T15:04:45.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/167/17' 2024-03-28T15:04:45.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/167' 2024-03-28T15:04:45.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/140/19' 2024-03-28T15:04:45.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/140/15' 2024-03-28T15:04:45.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/140/20' 2024-03-28T15:04:45.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/140/12' 2024-03-28T15:04:45.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/140/10' 2024-03-28T15:04:45.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/140/11' 2024-03-28T15:04:45.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/140/16' 2024-03-28T15:04:45.650 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/140/14' 2024-03-28T15:04:45.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/140/18' 2024-03-28T15:04:45.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/140/13' 2024-03-28T15:04:45.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/140/17' 2024-03-28T15:04:45.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/140' 2024-03-28T15:04:45.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/142/19' 2024-03-28T15:04:45.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/142/15' 2024-03-28T15:04:45.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/142/20' 2024-03-28T15:04:45.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/142/12' 2024-03-28T15:04:45.651 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/142/10' 2024-03-28T15:04:46.125 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/142/11' 2024-03-28T15:04:46.125 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/142/16' 2024-03-28T15:04:46.125 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/142/14' 2024-03-28T15:04:46.125 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/142/18' 2024-03-28T15:04:46.125 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/142/13' 2024-03-28T15:04:46.125 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/142/17' 2024-03-28T15:04:46.125 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/142' 2024-03-28T15:04:46.125 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/150/19' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/150/15' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/150/20' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/150/12' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/150/10' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/150/11' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/150/16' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/150/14' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/150/18' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/150/13' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/150/17' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/150' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/105/19' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/105/15' 2024-03-28T15:04:46.126 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/105/20' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/105/12' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/105/10' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/105/11' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/105/16' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/105/14' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/105/18' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/105/13' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/105/17' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/105' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/149/19' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/149/15' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/149/20' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/149/12' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/149/10' 2024-03-28T15:04:46.127 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/149/11' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/149/16' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/149/14' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/149/18' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/149/13' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/149/17' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/149' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/109/19' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/109/15' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/109/20' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/109/12' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/109/10' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/109/11' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/109/16' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/109/14' 2024-03-28T15:04:46.128 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/109/18' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/109/13' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/109/17' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/109' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/158/19' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/158/15' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/158/20' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/158/12' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/158/10' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/158/11' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/158/16' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/158/14' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/158/18' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/158/13' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/158/17' 2024-03-28T15:04:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/158' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/106/19' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/106/15' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/106/20' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/106/12' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/106/10' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/106/11' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/106/16' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/106/14' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/106/18' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/106/13' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/106/17' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/106' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/189/19' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/189/15' 2024-03-28T15:04:46.130 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/189/20' 2024-03-28T15:04:46.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/189/12' 2024-03-28T15:04:46.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/189/10' 2024-03-28T15:04:46.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/189/11' 2024-03-28T15:04:46.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/189/16' 2024-03-28T15:04:46.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/189/14' 2024-03-28T15:04:46.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/189/18' 2024-03-28T15:04:46.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/189/13' 2024-03-28T15:04:46.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/189/17' 2024-03-28T15:04:46.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/189' 2024-03-28T15:04:46.131 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/133/19' 2024-03-28T15:04:46.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/133/15' 2024-03-28T15:04:46.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/133/20' 2024-03-28T15:04:46.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/133/12' 2024-03-28T15:04:46.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/133/10' 2024-03-28T15:04:46.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/133/11' 2024-03-28T15:04:46.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/133/16' 2024-03-28T15:04:46.132 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/133/14' 2024-03-28T15:04:46.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/133/18' 2024-03-28T15:04:46.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/133/13' 2024-03-28T15:04:46.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/133/17' 2024-03-28T15:04:46.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/133' 2024-03-28T15:04:46.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/173/19' 2024-03-28T15:04:46.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/173/15' 2024-03-28T15:04:46.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/173/20' 2024-03-28T15:04:46.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/173/12' 2024-03-28T15:04:46.133 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/173/10' 2024-03-28T15:04:46.134 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/173/11' 2024-03-28T15:04:46.466 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/173/16' 2024-03-28T15:04:46.466 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/173/14' 2024-03-28T15:04:46.466 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/173/18' 2024-03-28T15:04:46.466 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/173/13' 2024-03-28T15:04:46.466 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/173/17' 2024-03-28T15:04:46.467 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/173' 2024-03-28T15:04:46.467 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/174/19' 2024-03-28T15:04:46.467 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/174/15' 2024-03-28T15:04:46.467 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/174/20' 2024-03-28T15:04:46.467 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/174/12' 2024-03-28T15:04:46.467 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/174/10' 2024-03-28T15:04:46.467 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/174/11' 2024-03-28T15:04:46.467 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/174/16' 2024-03-28T15:04:46.467 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/174/14' 2024-03-28T15:04:46.467 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/174/18' 2024-03-28T15:04:46.467 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/174/13' 2024-03-28T15:04:46.468 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/174/17' 2024-03-28T15:04:46.469 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/174' 2024-03-28T15:04:46.469 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/155/19' 2024-03-28T15:04:46.469 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/155/15' 2024-03-28T15:04:46.470 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/155/20' 2024-03-28T15:04:46.470 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/155/12' 2024-03-28T15:04:46.470 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/155/10' 2024-03-28T15:04:46.471 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/155/11' 2024-03-28T15:04:46.471 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/155/16' 2024-03-28T15:04:46.471 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/155/14' 2024-03-28T15:04:46.471 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/155/18' 2024-03-28T15:04:46.471 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/155/13' 2024-03-28T15:04:46.471 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/155/17' 2024-03-28T15:04:46.472 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/155' 2024-03-28T15:04:46.472 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/114/19' 2024-03-28T15:04:46.472 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/114/15' 2024-03-28T15:04:46.472 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/114/20' 2024-03-28T15:04:46.472 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/114/12' 2024-03-28T15:04:46.472 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/114/10' 2024-03-28T15:04:46.472 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/114/11' 2024-03-28T15:04:46.473 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/114/16' 2024-03-28T15:04:46.473 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/114/14' 2024-03-28T15:04:46.473 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/114/18' 2024-03-28T15:04:46.473 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/114/13' 2024-03-28T15:04:46.473 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/114/17' 2024-03-28T15:04:46.473 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/114' 2024-03-28T15:04:46.474 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/159/19' 2024-03-28T15:04:46.474 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/159/15' 2024-03-28T15:04:46.474 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/159/20' 2024-03-28T15:04:46.474 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/159/12' 2024-03-28T15:04:46.474 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/159/10' 2024-03-28T15:04:46.474 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/159/11' 2024-03-28T15:04:46.475 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/159/16' 2024-03-28T15:04:46.475 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/159/14' 2024-03-28T15:04:46.475 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/159/18' 2024-03-28T15:04:46.475 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/159/13' 2024-03-28T15:04:46.475 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/159/17' 2024-03-28T15:04:46.476 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/159' 2024-03-28T15:04:46.476 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/187/19' 2024-03-28T15:04:46.476 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/187/15' 2024-03-28T15:04:46.476 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/187/20' 2024-03-28T15:04:46.476 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/187/12' 2024-03-28T15:04:46.476 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/187/10' 2024-03-28T15:04:46.476 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/187/11' 2024-03-28T15:04:46.477 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/187/16' 2024-03-28T15:04:46.477 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/187/14' 2024-03-28T15:04:46.477 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/187/18' 2024-03-28T15:04:46.477 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/187/13' 2024-03-28T15:04:46.477 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/187/17' 2024-03-28T15:04:46.477 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/187' 2024-03-28T15:04:46.478 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/185/19' 2024-03-28T15:04:46.478 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/185/15' 2024-03-28T15:04:46.478 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/185/20' 2024-03-28T15:04:46.478 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/185/12' 2024-03-28T15:04:46.478 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/185/10' 2024-03-28T15:04:46.478 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/185/11' 2024-03-28T15:04:46.479 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/185/16' 2024-03-28T15:04:46.479 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/185/14' 2024-03-28T15:04:46.479 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/185/18' 2024-03-28T15:04:46.479 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/185/13' 2024-03-28T15:04:46.479 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/185/17' 2024-03-28T15:04:46.479 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/185' 2024-03-28T15:04:46.479 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/120/19' 2024-03-28T15:04:46.479 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/120/15' 2024-03-28T15:04:46.479 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/120/20' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/120/12' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/120/10' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/120/11' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/120/16' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/120/14' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/120/18' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/120/13' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/120/17' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/120' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/110/19' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/110/15' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/110/20' 2024-03-28T15:04:46.480 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/110/12' 2024-03-28T15:04:46.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/110/10' 2024-03-28T15:04:46.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/110/11' 2024-03-28T15:04:46.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/110/16' 2024-03-28T15:04:46.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/110/14' 2024-03-28T15:04:46.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/110/18' 2024-03-28T15:04:46.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/110/13' 2024-03-28T15:04:46.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/110/17' 2024-03-28T15:04:46.481 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/110' 2024-03-28T15:04:46.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/137/19' 2024-03-28T15:04:46.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/137/15' 2024-03-28T15:04:46.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/137/20' 2024-03-28T15:04:46.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/137/12' 2024-03-28T15:04:46.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/137/10' 2024-03-28T15:04:46.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/137/11' 2024-03-28T15:04:46.482 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/137/16' 2024-03-28T15:04:46.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/137/14' 2024-03-28T15:04:46.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/137/18' 2024-03-28T15:04:46.853 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/137/13' 2024-03-28T15:04:46.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/137/17' 2024-03-28T15:04:46.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/137' 2024-03-28T15:04:46.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/198/19' 2024-03-28T15:04:46.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/198/15' 2024-03-28T15:04:46.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/198/20' 2024-03-28T15:04:46.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/198/12' 2024-03-28T15:04:46.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/198/10' 2024-03-28T15:04:46.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/198/11' 2024-03-28T15:04:46.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/198/16' 2024-03-28T15:04:46.854 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/198/14' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/198/18' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/198/13' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/198/17' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/198' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/136/19' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/136/15' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/136/20' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/136/12' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/136/10' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/136/11' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/136/16' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/136/14' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/136/18' 2024-03-28T15:04:46.855 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/136/13' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/136/17' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/136' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/125/19' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/125/15' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/125/20' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/125/12' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/125/10' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/125/11' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/125/16' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/125/14' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/125/18' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/125/13' 2024-03-28T15:04:46.856 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/125/17' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/125' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/194/19' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/194/15' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/194/20' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/194/12' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/194/10' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/194/11' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/194/16' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/194/14' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/194/18' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/194/13' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/194/17' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/194' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/119/19' 2024-03-28T15:04:46.857 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/119/15' 2024-03-28T15:04:46.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/119/20' 2024-03-28T15:04:46.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/119/12' 2024-03-28T15:04:46.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/119/10' 2024-03-28T15:04:46.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/119/11' 2024-03-28T15:04:46.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/119/16' 2024-03-28T15:04:46.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/119/14' 2024-03-28T15:04:46.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/119/18' 2024-03-28T15:04:46.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/119/13' 2024-03-28T15:04:46.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/119/17' 2024-03-28T15:04:46.858 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/119' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/157/19' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/157/15' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/157/20' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/157/12' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/157/10' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/157/11' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/157/16' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/157/14' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/157/18' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/157/13' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/157/17' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/157' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/172/19' 2024-03-28T15:04:46.859 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/172/15' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/172/20' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/172/12' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/172/10' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/172/11' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/172/16' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/172/14' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/172/18' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/172/13' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/172/17' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/172' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/186/19' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/186/15' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/186/20' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/186/12' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/186/10' 2024-03-28T15:04:46.860 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/186/11' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/186/16' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/186/14' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/186/18' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/186/13' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/186/17' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/186' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/190/19' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/190/15' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/190/20' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/190/12' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/190/10' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/190/11' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/190/16' 2024-03-28T15:04:46.861 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/190/14' 2024-03-28T15:04:47.172 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/190/18' 2024-03-28T15:04:47.172 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/190/13' 2024-03-28T15:04:47.172 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/190/17' 2024-03-28T15:04:47.172 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/190' 2024-03-28T15:04:47.172 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/104/19' 2024-03-28T15:04:47.172 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/104/15' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/104/20' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/104/12' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/104/10' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/104/11' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/104/16' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/104/14' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/104/18' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/104/13' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/104/17' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/104' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/164/19' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/164/15' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/164/20' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/164/12' 2024-03-28T15:04:47.173 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/164/10' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/164/11' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/164/16' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/164/14' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/164/18' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/164/13' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/164/17' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/164' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/112/19' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/112/15' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/112/20' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/112/12' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/112/10' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/112/11' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/112/16' 2024-03-28T15:04:47.174 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/112/14' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/112/18' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/112/13' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/112/17' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/112' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/100/19' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/100/15' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/100/20' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/100/12' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/100/10' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/100/11' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/100/16' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/100/14' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/100/18' 2024-03-28T15:04:47.175 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/100/13' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/100/17' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/100' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/134/19' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/134/15' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/134/20' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/134/12' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/134/10' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/134/11' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/134/16' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/134/14' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/134/18' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/134/13' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/134/17' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/134' 2024-03-28T15:04:47.176 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/115/19' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/115/15' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/115/20' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/115/12' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/115/10' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/115/11' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/115/16' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/115/14' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/115/18' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/115/13' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/115/17' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/115' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/116/19' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/116/15' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/116/20' 2024-03-28T15:04:47.177 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/116/12' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/116/10' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/116/11' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/116/16' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/116/14' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/116/18' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/116/13' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/116/17' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/116' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/195/19' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/195/15' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/195/20' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/195/12' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/195/10' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/195/11' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/195/16' 2024-03-28T15:04:47.178 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/195/14' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/195/18' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/195/13' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/195/17' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/195' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/138/19' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/138/15' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/138/20' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/138/12' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/138/10' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/138/11' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/138/16' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/138/14' 2024-03-28T15:04:47.179 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/138/18' 2024-03-28T15:04:47.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/138/13' 2024-03-28T15:04:47.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/138/17' 2024-03-28T15:04:47.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/138' 2024-03-28T15:04:47.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/135/19' 2024-03-28T15:04:47.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/135/15' 2024-03-28T15:04:47.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/135/20' 2024-03-28T15:04:47.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/135/12' 2024-03-28T15:04:47.493 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/135/10' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/135/11' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/135/16' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/135/14' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/135/18' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/135/13' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/135/17' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/135' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/117/19' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/117/15' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/117/20' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/117/12' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/117/10' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/117/11' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/117/16' 2024-03-28T15:04:47.494 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/117/14' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/117/18' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/117/13' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/117/17' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/117' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/143/19' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/143/15' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/143/20' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/143/12' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/143/10' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/143/11' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/143/16' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/143/14' 2024-03-28T15:04:47.495 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/143/18' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/143/13' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/143/17' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/143' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/180/19' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/180/15' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/180/20' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/180/12' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/180/10' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/180/11' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/180/16' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/180/14' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/180/18' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/180/13' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/180/17' 2024-03-28T15:04:47.496 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/180' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/126/19' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/126/15' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/126/20' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/126/12' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/126/10' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/126/11' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/126/16' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/126/14' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/126/18' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/126/13' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/126/17' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/126' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/161/19' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/161/15' 2024-03-28T15:04:47.497 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/161/20' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/161/12' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/161/10' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/161/11' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/161/16' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/161/14' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/161/18' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/161/13' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/161/17' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/161' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/176/19' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/176/15' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/176/20' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/176/12' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/176/10' 2024-03-28T15:04:47.498 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/176/11' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/176/16' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/176/14' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/176/18' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/176/13' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/176/17' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/176' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/111/19' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/111/15' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/111/20' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/111/12' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/111/10' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/111/11' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/111/16' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/111/14' 2024-03-28T15:04:47.499 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/111/18' 2024-03-28T15:04:47.500 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/111/13' 2024-03-28T15:04:47.500 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/111/17' 2024-03-28T15:04:47.500 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/111' 2024-03-28T15:04:47.500 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/191/19' 2024-03-28T15:04:47.500 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/191/15' 2024-03-28T15:04:47.500 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/191/20' 2024-03-28T15:04:47.500 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/191/12' 2024-03-28T15:04:47.500 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/191/10' 2024-03-28T15:04:47.500 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/191/11' 2024-03-28T15:04:47.500 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/191/16' 2024-03-28T15:04:47.500 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/191/14' 2024-03-28T15:04:47.501 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/191/18' 2024-03-28T15:04:47.501 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/191/13' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/191/17' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/191' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/184/19' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/184/15' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/184/20' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/184/12' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/184/10' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/184/11' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/184/16' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/184/14' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/184/18' 2024-03-28T15:04:47.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/184/13' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/184/17' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/184' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/192/19' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/192/15' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/192/20' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/192/12' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/192/10' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/192/11' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/192/16' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/192/14' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/192/18' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/192/13' 2024-03-28T15:04:47.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/192/17' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/192' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/103/19' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/103/15' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/103/20' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/103/12' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/103/10' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/103/11' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/103/16' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/103/14' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/103/18' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/103/13' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/103/17' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/103' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/128/19' 2024-03-28T15:04:47.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/128/15' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/128/20' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/128/12' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/128/10' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/128/11' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/128/16' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/128/14' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/128/18' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/128/13' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/128/17' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/128' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/144/19' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/144/15' 2024-03-28T15:04:47.916 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/144/20' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/144/12' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/144/10' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/144/11' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/144/16' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/144/14' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/144/18' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/144/13' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/144/17' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/144' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/146/19' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/146/15' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/146/20' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/146/12' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/146/10' 2024-03-28T15:04:47.917 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/146/11' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/146/16' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/146/14' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/146/18' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/146/13' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/146/17' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/146' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/141/19' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/141/15' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/141/20' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/141/12' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/141/10' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/141/11' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/141/16' 2024-03-28T15:04:47.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/141/14' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/141/18' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/141/13' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/141/17' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/141' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/182/19' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/182/15' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/182/20' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/182/12' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/182/10' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/182/11' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/182/16' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/182/14' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/182/18' 2024-03-28T15:04:47.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/182/13' 2024-03-28T15:04:47.920 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/182/17' 2024-03-28T15:04:47.920 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/182' 2024-03-28T15:04:47.920 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/107/19' 2024-03-28T15:04:47.920 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/107/15' 2024-03-28T15:04:47.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/107/20' 2024-03-28T15:04:47.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/107/12' 2024-03-28T15:04:47.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/107/10' 2024-03-28T15:04:47.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/107/11' 2024-03-28T15:04:47.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/107/16' 2024-03-28T15:04:47.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/107/14' 2024-03-28T15:04:47.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/107/18' 2024-03-28T15:04:47.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/107/13' 2024-03-28T15:04:47.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/107/17' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/107' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/178/19' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/178/15' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/178/20' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/178/12' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/178/10' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/178/11' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/178/16' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/178/14' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/178/18' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/178/13' 2024-03-28T15:04:48.206 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/178/17' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/178' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/156/19' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/156/15' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/156/20' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/156/12' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/156/10' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/156/11' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/156/16' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/156/14' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/156/18' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/156/13' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/156/17' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/156' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/131/19' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/131/15' 2024-03-28T15:04:48.207 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/131/20' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/131/12' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/131/10' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/131/11' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/131/16' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/131/14' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/131/18' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/131/13' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/131/17' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/131' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/123/19' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/123/15' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/123/20' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/123/12' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/123/10' 2024-03-28T15:04:48.208 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/123/11' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/123/16' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/123/14' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/123/18' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/123/13' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/123/17' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/123' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/160/19' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/160/15' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/160/20' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/160/12' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/160/10' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/160/11' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/160/16' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/160/14' 2024-03-28T15:04:48.209 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/160/18' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/160/13' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/160/17' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/160' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/165/19' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/165/15' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/165/20' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/165/12' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/165/10' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/165/11' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/165/16' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/165/14' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/165/18' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/165/13' 2024-03-28T15:04:48.210 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/165/17' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/165' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/101/19' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/101/15' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/101/20' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/101/12' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/101/10' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/101/11' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/101/16' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/101/14' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/101/18' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/101/13' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/101/17' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/101' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/148/19' 2024-03-28T15:04:48.211 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/148/15' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/148/20' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/148/12' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/148/10' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/148/11' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/148/16' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/148/14' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/148/18' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/148/13' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/148/17' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/148' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/168/19' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/168/15' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/168/20' 2024-03-28T15:04:48.212 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/168/12' 2024-03-28T15:04:48.213 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/168/10' 2024-03-28T15:04:48.213 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/168/11' 2024-03-28T15:04:48.213 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/168/16' 2024-03-28T15:04:48.213 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/168/14' 2024-03-28T15:04:48.213 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/168/18' 2024-03-28T15:04:48.213 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/168/13' 2024-03-28T15:04:48.213 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/168/17' 2024-03-28T15:04:48.213 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/168' 2024-03-28T15:04:48.541 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/179/19' 2024-03-28T15:04:48.541 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/179/15' 2024-03-28T15:04:48.541 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/179/20' 2024-03-28T15:04:48.541 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/179/12' 2024-03-28T15:04:48.541 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/179/10' 2024-03-28T15:04:48.541 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/179/11' 2024-03-28T15:04:48.541 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/179/16' 2024-03-28T15:04:48.541 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/179/14' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/179/18' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/179/13' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/179/17' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/179' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/132/19' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/132/15' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/132/20' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/132/12' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/132/10' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/132/11' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/132/16' 2024-03-28T15:04:48.542 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/132/14' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/132/18' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/132/13' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/132/17' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/132' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/199/19' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/199/15' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/199/20' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/199/12' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/199/10' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/199/11' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/199/16' 2024-03-28T15:04:48.543 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/199/14' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/199/18' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/199/13' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/199/17' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/199' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/163/19' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/163/15' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/163/20' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/163/12' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/163/10' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/163/11' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/163/16' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/163/14' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/163/18' 2024-03-28T15:04:48.544 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/163/13' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/163/17' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/163' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/153/19' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/153/15' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/153/20' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/153/12' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/153/10' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/153/11' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/153/16' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/153/14' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/153/18' 2024-03-28T15:04:48.545 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/153/13' 2024-03-28T15:04:48.546 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/153/17' 2024-03-28T15:04:48.546 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/153' 2024-03-28T15:04:48.546 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/122/19' 2024-03-28T15:04:48.546 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/122/15' 2024-03-28T15:04:48.546 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/122/20' 2024-03-28T15:04:48.546 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/122/12' 2024-03-28T15:04:48.546 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/122/10' 2024-03-28T15:04:48.546 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/122/11' 2024-03-28T15:04:48.546 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/122/16' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/122/14' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/122/18' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/122/13' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/122/17' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/122' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/162/19' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/162/15' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/162/20' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/162/12' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/162/10' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/162/11' 2024-03-28T15:04:48.547 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/162/16' 2024-03-28T15:04:48.548 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/162/14' 2024-03-28T15:04:48.548 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/162/18' 2024-03-28T15:04:48.548 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/162/13' 2024-03-28T15:04:48.548 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/162/17' 2024-03-28T15:04:48.548 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/162' 2024-03-28T15:04:48.548 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/197/19' 2024-03-28T15:04:48.548 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/197/15' 2024-03-28T15:04:48.548 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/197/20' 2024-03-28T15:04:48.548 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/197/12' 2024-03-28T15:04:48.548 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/197/10' 2024-03-28T15:04:48.548 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/197/11' 2024-03-28T15:04:48.549 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/197/16' 2024-03-28T15:04:48.549 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/197/14' 2024-03-28T15:04:48.549 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/197/18' 2024-03-28T15:04:48.549 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/197/13' 2024-03-28T15:04:48.549 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/197/17' 2024-03-28T15:04:48.549 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/197' 2024-03-28T15:04:48.549 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/113/19' 2024-03-28T15:04:48.549 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/113/15' 2024-03-28T15:04:48.549 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/113/20' 2024-03-28T15:04:48.549 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/113/12' 2024-03-28T15:04:48.550 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/113/10' 2024-03-28T15:04:48.550 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/113/11' 2024-03-28T15:04:48.550 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/113/16' 2024-03-28T15:04:48.550 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/113/14' 2024-03-28T15:04:48.550 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/113/18' 2024-03-28T15:04:48.550 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/113/13' 2024-03-28T15:04:48.550 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/113/17' 2024-03-28T15:04:48.550 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/113' 2024-03-28T15:04:48.550 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/152/19' 2024-03-28T15:04:48.909 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/152/15' 2024-03-28T15:04:48.909 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/152/20' 2024-03-28T15:04:48.909 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/152/12' 2024-03-28T15:04:48.909 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/152/10' 2024-03-28T15:04:48.909 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/152/11' 2024-03-28T15:04:48.909 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/152/16' 2024-03-28T15:04:48.909 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/152/14' 2024-03-28T15:04:48.909 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/152/18' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/152/13' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/152/17' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/152' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/121/19' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/121/15' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/121/20' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/121/12' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/121/10' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/121/11' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/121/16' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/121/14' 2024-03-28T15:04:48.910 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/121/18' 2024-03-28T15:04:48.911 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/121/13' 2024-03-28T15:04:48.911 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/121/17' 2024-03-28T15:04:48.911 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/121' 2024-03-28T15:04:48.911 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/118/19' 2024-03-28T15:04:48.911 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/118/15' 2024-03-28T15:04:48.911 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/118/20' 2024-03-28T15:04:48.911 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/118/12' 2024-03-28T15:04:48.911 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/118/10' 2024-03-28T15:04:48.911 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/118/11' 2024-03-28T15:04:48.911 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/118/16' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/118/14' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/118/18' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/118/13' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/118/17' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/118' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/196/19' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/196/15' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/196/20' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/196/12' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/196/10' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/196/11' 2024-03-28T15:04:48.912 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/196/16' 2024-03-28T15:04:48.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/196/14' 2024-03-28T15:04:48.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/196/18' 2024-03-28T15:04:48.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/196/13' 2024-03-28T15:04:48.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/196/17' 2024-03-28T15:04:48.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/196' 2024-03-28T15:04:48.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/124/19' 2024-03-28T15:04:48.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/124/15' 2024-03-28T15:04:48.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/124/20' 2024-03-28T15:04:48.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/124/12' 2024-03-28T15:04:48.913 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/124/10' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/124/11' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/124/16' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/124/14' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/124/18' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/124/13' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/124/17' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/124' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/188/19' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/188/15' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/188/20' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/188/12' 2024-03-28T15:04:48.914 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/188/10' 2024-03-28T15:04:48.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/188/11' 2024-03-28T15:04:48.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/188/16' 2024-03-28T15:04:48.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/188/14' 2024-03-28T15:04:48.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/188/18' 2024-03-28T15:04:48.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/188/13' 2024-03-28T15:04:48.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/188/17' 2024-03-28T15:04:48.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/188' 2024-03-28T15:04:48.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/145/19' 2024-03-28T15:04:48.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/145/15' 2024-03-28T15:04:48.915 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/145/20' 2024-03-28T15:04:48.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/145/12' 2024-03-28T15:04:48.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/145/10' 2024-03-28T15:04:48.918 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/145/11' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/145/16' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/145/14' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/145/18' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/145/13' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/145/17' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/145' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/130/19' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/130/15' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/130/20' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/130/12' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/130/10' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/130/11' 2024-03-28T15:04:48.919 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/130/16' 2024-03-28T15:04:48.920 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/130/14' 2024-03-28T15:04:48.920 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/130/18' 2024-03-28T15:04:48.920 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/130/13' 2024-03-28T15:04:48.920 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/130/17' 2024-03-28T15:04:48.920 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/130' 2024-03-28T15:04:48.920 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/193/19' 2024-03-28T15:04:48.920 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/193/15' 2024-03-28T15:04:48.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/193/20' 2024-03-28T15:04:48.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/193/12' 2024-03-28T15:04:48.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/193/10' 2024-03-28T15:04:48.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/193/11' 2024-03-28T15:04:48.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/193/16' 2024-03-28T15:04:48.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/193/14' 2024-03-28T15:04:48.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/193/18' 2024-03-28T15:04:48.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/193/13' 2024-03-28T15:04:48.921 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/193/17' 2024-03-28T15:04:48.922 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/193' 2024-03-28T15:04:48.922 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/175/19' 2024-03-28T15:04:48.922 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/175/15' 2024-03-28T15:04:49.194 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/175/20' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/175/12' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/175/10' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/175/11' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/175/16' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/175/14' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/175/18' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/175/13' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/175/17' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/175' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/181/19' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/181/15' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/181/20' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/181/12' 2024-03-28T15:04:49.195 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/181/10' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/181/11' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/181/16' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/181/14' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/181/18' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/181/13' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/181/17' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/181' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/127/19' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/127/15' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/127/20' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/127/12' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/127/10' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/127/11' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/127/16' 2024-03-28T15:04:49.196 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/127/14' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/127/18' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/127/13' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/127/17' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/127' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/108/19' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/108/15' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/108/20' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/108/12' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/108/10' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/108/11' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/108/16' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/108/14' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/108/18' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/108/13' 2024-03-28T15:04:49.197 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/108/17' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/108' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/183/19' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/183/15' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/183/20' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/183/12' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/183/10' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/183/11' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/183/16' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/183/14' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/183/18' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/183/13' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/183/17' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/183' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/177/19' 2024-03-28T15:04:49.198 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/177/15' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/177/20' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/177/12' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/177/10' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/177/11' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/177/16' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/177/14' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/177/18' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/177/13' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/177/17' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/177' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/169/19' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/169/15' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/169/20' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/169/12' 2024-03-28T15:04:49.199 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/169/10' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/169/11' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/169/16' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/169/14' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/169/18' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/169/13' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/169/17' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/169' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/154/19' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/154/15' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/154/20' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/154/12' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/154/10' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/154/11' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/154/16' 2024-03-28T15:04:49.200 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/154/14' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/154/18' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/154/13' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/154/17' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/154' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/171/19' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/171/15' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/171/20' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/171/12' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/171/10' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/171/11' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/171/16' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/171/14' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/171/18' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/171/13' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/171/17' 2024-03-28T15:04:49.201 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/171' 2024-03-28T15:04:49.202 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/139/19' 2024-03-28T15:04:49.202 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/139/15' 2024-03-28T15:04:49.202 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/139/20' 2024-03-28T15:04:49.304 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/139/12' 2024-03-28T15:04:49.304 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/139/10' 2024-03-28T15:04:49.304 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/139/11' 2024-03-28T15:04:49.304 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/139/16' 2024-03-28T15:04:49.305 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/139/14' 2024-03-28T15:04:49.305 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/139/18' 2024-03-28T15:04:49.305 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/139/13' 2024-03-28T15:04:49.305 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/139/17' 2024-03-28T15:04:49.305 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/139' 2024-03-28T15:04:49.306 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/102/19' 2024-03-28T15:04:49.306 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/102/15' 2024-03-28T15:04:49.306 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/102/20' 2024-03-28T15:04:49.306 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/102/12' 2024-03-28T15:04:49.306 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/102/10' 2024-03-28T15:04:49.307 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/102/11' 2024-03-28T15:04:49.307 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/102/16' 2024-03-28T15:04:49.307 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/102/14' 2024-03-28T15:04:49.307 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/102/18' 2024-03-28T15:04:49.307 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/102/13' 2024-03-28T15:04:49.308 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/102/17' 2024-03-28T15:04:49.308 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/102' 2024-03-28T15:04:49.308 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/129/19' 2024-03-28T15:04:49.308 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/129/15' 2024-03-28T15:04:49.309 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/129/20' 2024-03-28T15:04:49.309 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/129/12' 2024-03-28T15:04:49.309 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/129/10' 2024-03-28T15:04:49.309 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/129/11' 2024-03-28T15:04:49.309 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/129/16' 2024-03-28T15:04:49.310 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/129/14' 2024-03-28T15:04:49.310 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/129/18' 2024-03-28T15:04:49.310 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/129/13' 2024-03-28T15:04:49.310 INFO:tasks.workunit.client.0.smithi186.stdout:removed './copyofsnap1/test1/129/17' 2024-03-28T15:04:49.311 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1/129' 2024-03-28T15:04:49.311 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1/test1' 2024-03-28T15:04:49.311 INFO:tasks.workunit.client.0.smithi186.stdout:removed directory './copyofsnap1' 2024-03-28T15:04:49.311 INFO:tasks.workunit.client.0.smithi186.stdout:OK 2024-03-28T15:04:49.312 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T15:04:49.312 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:04:49.349 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2024-03-28T15:04:49.349 DEBUG:teuthology.orchestra.run.smithi186:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2024-03-28T15:04:49.412 INFO:tasks.workunit.client.0.smithi186.stderr:+ set -e 2024-03-28T15:04:49.413 INFO:tasks.workunit.client.0.smithi186.stderr:+ touch foo 2024-03-28T15:04:49.414 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod +x foo 2024-03-28T15:04:49.415 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir .snap/s 2024-03-28T15:04:49.428 INFO:tasks.workunit.client.0.smithi186.stderr:+ find .snap/s/foo -executable 2024-03-28T15:04:49.428 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep foo 2024-03-28T15:04:49.433 INFO:tasks.workunit.client.0.smithi186.stdout:.snap/s/foo 2024-03-28T15:04:49.433 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir .snap/s 2024-03-28T15:04:49.439 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm foo 2024-03-28T15:04:49.441 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo OK 2024-03-28T15:04:49.442 INFO:tasks.workunit.client.0.smithi186.stdout:OK 2024-03-28T15:04:49.442 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T15:04:49.442 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:04:49.513 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2024-03-28T15:04:49.514 DEBUG:teuthology.orchestra.run.smithi186:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2024-03-28T15:04:49.577 INFO:tasks.workunit.client.0.smithi186.stderr:+ set -e 2024-03-28T15:04:49.578 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir foo 2024-03-28T15:04:49.580 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo x 2024-03-28T15:04:49.582 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:50.079 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir foo/.snap/ss 2024-03-28T15:04:50.118 INFO:tasks.workunit.client.0.smithi186.stderr:+ ln foo/x foo/xx 2024-03-28T15:04:50.121 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat foo/.snap/ss/x 2024-03-28T15:04:50.127 INFO:tasks.workunit.client.0.smithi186.stdout:x 2024-03-28T15:04:50.127 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir foo/.snap/ss 2024-03-28T15:04:50.133 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo a 2024-03-28T15:04:50.135 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo b 2024-03-28T15:04:50.137 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir foo/.snap/s 2024-03-28T15:04:50.145 INFO:tasks.workunit.client.0.smithi186.stderr:++ cat foo/.snap/s/a 2024-03-28T15:04:50.153 INFO:tasks.workunit.client.0.smithi186.stderr:+ r=a 2024-03-28T15:04:50.153 INFO:tasks.workunit.client.0.smithi186.stderr:+ '[' -z a ']' 2024-03-28T15:04:50.153 INFO:tasks.workunit.client.0.smithi186.stderr:+ ln foo/b foo/b2 2024-03-28T15:04:50.157 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat foo/.snap/s/b 2024-03-28T15:04:50.162 INFO:tasks.workunit.client.0.smithi186.stdout:b 2024-03-28T15:04:50.162 INFO:tasks.workunit.client.0.smithi186.stdout:this used to hang: 2024-03-28T15:04:50.163 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 'this used to hang:' 2024-03-28T15:04:50.163 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo more 2024-03-28T15:04:50.165 INFO:tasks.workunit.client.0.smithi186.stdout:oh, it didn't hang! good job. 2024-03-28T15:04:50.165 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2024-03-28T15:04:50.165 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat foo/b 2024-03-28T15:04:50.166 INFO:tasks.workunit.client.0.smithi186.stdout:b 2024-03-28T15:04:50.166 INFO:tasks.workunit.client.0.smithi186.stdout:more 2024-03-28T15:04:50.167 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir foo/.snap/s 2024-03-28T15:04:50.174 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm -r foo 2024-03-28T15:04:50.198 INFO:tasks.workunit.client.0.smithi186.stdout:OK 2024-03-28T15:04:50.199 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo OK 2024-03-28T15:04:50.199 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T15:04:50.199 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:04:50.275 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2024-03-28T15:04:50.275 DEBUG:teuthology.orchestra.run.smithi186:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2024-03-28T15:04:50.339 INFO:tasks.workunit.client.0.smithi186.stderr:+ set -e 2024-03-28T15:04:50.339 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:50.341 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir .snap/bar 2024-03-28T15:04:50.349 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/foo 2024-03-28T15:04:50.355 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/foo 2024-03-28T15:04:50.361 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:50.364 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/bar/a 2024-03-28T15:04:50.370 INFO:tasks.workunit.client.0.smithi186.stdout: File: .snap/bar/a 2024-03-28T15:04:50.370 INFO:tasks.workunit.client.0.smithi186.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-03-28T15:04:50.370 INFO:tasks.workunit.client.0.smithi186.stdout:Device: 2bh/43d Inode: 845524441765765 Links: 2 2024-03-28T15:04:50.370 INFO:tasks.workunit.client.0.smithi186.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-28T15:04:50.370 INFO:tasks.workunit.client.0.smithi186.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-28T15:04:50.370 INFO:tasks.workunit.client.0.smithi186.stdout:Access: 2024-03-28 15:04:50.341008862 +0000 2024-03-28T15:04:50.370 INFO:tasks.workunit.client.0.smithi186.stdout:Modify: 2024-03-28 15:04:50.341008862 +0000 2024-03-28T15:04:50.370 INFO:tasks.workunit.client.0.smithi186.stdout:Change: 2024-03-28 15:04:50.341008862 +0000 2024-03-28T15:04:50.370 INFO:tasks.workunit.client.0.smithi186.stdout: Birth: - 2024-03-28T15:04:50.370 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir .snap/bar 2024-03-28T15:04:50.375 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo OK 2024-03-28T15:04:50.376 INFO:tasks.workunit.client.0.smithi186.stdout:OK 2024-03-28T15:04:50.376 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T15:04:50.376 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:04:50.448 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2024-03-28T15:04:50.448 DEBUG:teuthology.orchestra.run.smithi186:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2024-03-28T15:04:50.511 INFO:tasks.workunit.client.0.smithi186.stderr:+ set -e 2024-03-28T15:04:50.511 INFO:tasks.workunit.client.0.smithi186.stderr:++ seq 1 20 2024-03-28T15:04:50.512 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:50.512 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:50.514 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:50.514 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:50.523 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:50.529 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:50.531 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:50.621 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:50.674 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:50.674 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:50.674 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:50.674 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:50.674 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:50.675 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:50.675 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:50.675 INFO:tasks.workunit.client.0.smithi186.stdout: 51302 ? 00:00:00 ps 2024-03-28T15:04:50.676 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:50.687 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:50.689 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:50.693 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:50.693 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:50.696 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:50.696 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:50.704 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:50.709 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:50.710 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:50.770 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:50.824 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:50.824 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:50.824 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:50.824 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:50.824 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:50.824 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:50.824 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:50.824 INFO:tasks.workunit.client.0.smithi186.stdout: 51312 ? 00:00:00 ps 2024-03-28T15:04:50.826 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:50.864 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:50.866 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:50.871 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:50.871 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:50.873 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:50.874 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:50.882 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:50.887 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:50.888 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:50.995 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:51.041 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:51.041 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:51.041 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:51.041 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:51.042 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:51.042 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:51.042 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:51.042 INFO:tasks.workunit.client.0.smithi186.stdout: 51322 ? 00:00:00 ps 2024-03-28T15:04:51.044 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:51.055 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:51.058 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:51.063 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:51.063 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:51.065 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:51.065 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:51.073 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:51.079 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:51.080 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:51.145 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:51.190 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:51.190 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:51.190 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:51.190 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:51.190 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:51.190 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:51.191 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:51.191 INFO:tasks.workunit.client.0.smithi186.stdout: 51332 ? 00:00:00 ps 2024-03-28T15:04:51.192 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:51.206 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:51.212 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:51.225 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:51.226 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:51.231 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:51.231 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:51.248 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:51.253 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:51.254 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:51.346 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:51.393 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:51.393 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:51.393 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:51.393 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:51.393 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:51.393 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:51.393 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:51.393 INFO:tasks.workunit.client.0.smithi186.stdout: 51342 ? 00:00:00 ps 2024-03-28T15:04:51.395 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:51.407 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:51.409 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:51.414 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:51.414 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:51.416 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:51.416 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:51.426 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:51.431 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:51.432 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:51.520 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:51.570 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:51.570 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:51.570 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:51.570 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:51.570 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:51.570 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:51.570 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:51.570 INFO:tasks.workunit.client.0.smithi186.stdout: 51352 ? 00:00:00 ps 2024-03-28T15:04:51.572 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:51.585 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:51.587 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:51.592 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:51.592 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:51.594 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:51.594 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:51.604 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:51.609 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:51.611 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:51.689 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:51.734 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:51.734 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:51.734 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:51.734 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:51.735 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:51.735 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:51.735 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:51.735 INFO:tasks.workunit.client.0.smithi186.stdout: 51362 ? 00:00:00 ps 2024-03-28T15:04:51.737 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:51.755 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:51.757 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:51.762 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:51.762 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:51.764 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:51.764 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:51.774 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:51.779 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:51.780 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:51.838 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:51.884 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:51.884 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:51.884 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:51.884 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:51.884 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:51.884 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:51.884 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:51.884 INFO:tasks.workunit.client.0.smithi186.stdout: 51372 ? 00:00:00 ps 2024-03-28T15:04:51.886 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:51.897 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:51.900 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:51.905 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:51.905 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:51.907 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:51.907 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:51.917 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:51.922 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:51.923 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:51.971 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:52.018 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:52.019 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:52.019 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:52.019 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:52.019 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:52.019 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:52.019 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:52.019 INFO:tasks.workunit.client.0.smithi186.stdout: 51382 ? 00:00:00 ps 2024-03-28T15:04:52.021 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:52.032 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:52.034 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:52.040 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:52.040 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:52.041 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:52.041 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:52.049 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:52.054 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:52.055 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:52.121 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:52.168 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:52.168 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:52.168 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:52.168 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:52.168 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:52.168 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:52.168 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:52.168 INFO:tasks.workunit.client.0.smithi186.stdout: 51392 ? 00:00:00 ps 2024-03-28T15:04:52.171 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:52.184 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:52.186 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:52.191 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:52.191 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:52.194 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:52.194 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:52.203 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:52.211 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:52.212 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:52.281 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:52.348 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:52.349 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:52.349 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:52.349 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:52.349 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:52.349 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:52.349 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:52.349 INFO:tasks.workunit.client.0.smithi186.stdout: 51402 ? 00:00:00 ps 2024-03-28T15:04:52.352 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:52.366 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:52.369 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:52.375 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:52.375 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:52.378 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:52.378 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:52.388 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:52.400 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:52.401 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:52.539 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:52.586 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:52.586 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:52.586 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:52.586 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:52.586 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:52.586 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:52.587 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:52.587 INFO:tasks.workunit.client.0.smithi186.stdout: 51412 ? 00:00:00 ps 2024-03-28T15:04:52.590 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:52.603 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:52.605 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:52.609 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:52.609 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:52.611 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:52.612 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:52.620 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:52.625 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:52.626 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:52.673 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:52.720 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:52.720 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:52.720 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:52.720 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:52.720 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:52.720 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:52.720 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:52.720 INFO:tasks.workunit.client.0.smithi186.stdout: 51422 ? 00:00:00 ps 2024-03-28T15:04:52.722 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:52.734 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:52.736 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:52.741 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:52.741 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:52.744 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:52.744 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:52.754 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:52.760 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:52.761 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:52.823 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:52.868 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:52.868 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:52.869 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:52.869 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:52.869 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:52.869 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:52.869 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:52.869 INFO:tasks.workunit.client.0.smithi186.stdout: 51432 ? 00:00:00 ps 2024-03-28T15:04:52.871 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:52.883 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:52.885 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:52.890 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:52.890 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:52.892 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:52.893 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:52.903 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:52.909 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:52.910 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:52.973 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:53.019 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:53.019 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:53.019 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:53.019 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:53.020 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:53.020 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:53.020 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:53.020 INFO:tasks.workunit.client.0.smithi186.stdout: 51442 ? 00:00:00 ps 2024-03-28T15:04:53.023 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:53.035 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:53.038 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:53.042 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:53.042 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:53.044 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:53.045 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:53.053 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:53.060 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:53.061 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:53.107 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:53.153 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:53.153 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:53.153 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:53.153 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:53.153 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:53.153 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:53.154 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:53.154 INFO:tasks.workunit.client.0.smithi186.stdout: 51452 ? 00:00:00 ps 2024-03-28T15:04:53.156 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:53.167 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:53.169 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:53.174 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:53.174 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:53.176 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:53.176 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:53.186 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:53.192 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:53.193 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:53.273 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:53.320 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:53.320 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:53.320 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:53.320 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:53.320 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:53.320 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:53.320 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:53.320 INFO:tasks.workunit.client.0.smithi186.stdout: 51462 ? 00:00:00 ps 2024-03-28T15:04:53.323 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:53.350 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:53.354 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:53.363 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:53.363 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:53.367 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:53.367 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:53.379 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:53.385 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:53.386 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:53.549 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:53.599 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:53.599 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:53.599 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:53.599 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:53.599 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:53.599 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:53.599 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:53.599 INFO:tasks.workunit.client.0.smithi186.stdout: 51472 ? 00:00:00 ps 2024-03-28T15:04:53.601 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:53.613 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:53.616 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:53.620 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:53.620 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:53.622 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:53.622 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:53.631 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:53.637 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:53.638 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:53.715 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:53.777 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:53.777 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:53.777 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:53.777 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:53.777 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:53.777 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:53.777 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:53.777 INFO:tasks.workunit.client.0.smithi186.stdout: 51482 ? 00:00:00 ps 2024-03-28T15:04:53.781 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:53.797 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:53.799 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:53.803 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 20` 2024-03-28T15:04:53.803 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a 2024-03-28T15:04:53.805 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/one 2024-03-28T15:04:53.805 INFO:tasks.workunit.client.0.smithi186.stderr:+ cat 2024-03-28T15:04:53.815 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir a/.snap/two 2024-03-28T15:04:53.820 INFO:tasks.workunit.client.0.smithi186.stderr:+ chmod 777 a/foo 2024-03-28T15:04:53.822 INFO:tasks.workunit.client.0.smithi186.stderr:+ sync 2024-03-28T15:04:53.899 INFO:tasks.workunit.client.0.smithi186.stderr:+ ps 2024-03-28T15:04:53.944 INFO:tasks.workunit.client.0.smithi186.stdout: PID TTY TIME CMD 2024-03-28T15:04:53.944 INFO:tasks.workunit.client.0.smithi186.stdout: 4843 ? 00:00:00 systemd 2024-03-28T15:04:53.944 INFO:tasks.workunit.client.0.smithi186.stdout: 4845 ? 00:00:00 (sd-pam) 2024-03-28T15:04:53.944 INFO:tasks.workunit.client.0.smithi186.stdout: 23484 ? 00:00:00 sshd 2024-03-28T15:04:53.944 INFO:tasks.workunit.client.0.smithi186.stdout: 46457 ? 00:00:00 bash 2024-03-28T15:04:53.945 INFO:tasks.workunit.client.0.smithi186.stdout: 51276 ? 00:00:00 timeout 2024-03-28T15:04:53.945 INFO:tasks.workunit.client.0.smithi186.stdout: 51294 ? 00:00:00 snaptest-double 2024-03-28T15:04:53.945 INFO:tasks.workunit.client.0.smithi186.stdout: 51492 ? 00:00:00 ps 2024-03-28T15:04:53.946 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T15:04:53.959 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm a/foo 2024-03-28T15:04:53.962 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir a 2024-03-28T15:04:53.967 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo OK 2024-03-28T15:04:53.968 INFO:tasks.workunit.client.0.smithi186.stdout:OK 2024-03-28T15:04:53.968 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T15:04:53.968 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:04:54.042 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2024-03-28T15:04:54.042 DEBUG:teuthology.orchestra.run.smithi186:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2024-03-28T15:04:54.105 INFO:tasks.workunit.client.0.smithi186.stderr:+ mkdir .snap/foo 2024-03-28T15:04:54.113 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2024-03-28T15:04:54.113 INFO:tasks.workunit.client.0.smithi186.stdout:We want ENOENT, not ESTALE, here. 2024-03-28T15:04:54.114 INFO:tasks.workunit.client.0.smithi186.stderr:++ seq 1 100 2024-03-28T15:04:54.114 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.115 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/1 2024-03-28T15:04:54.115 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.117 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2024-03-28T15:04:54.117 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.117 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/2 2024-03-28T15:04:54.118 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.119 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2024-03-28T15:04:54.120 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.120 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/3 2024-03-28T15:04:54.120 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.122 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2024-03-28T15:04:54.122 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.122 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/4 2024-03-28T15:04:54.123 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.124 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2024-03-28T15:04:54.125 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.125 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/5 2024-03-28T15:04:54.125 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.127 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2024-03-28T15:04:54.127 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.127 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/6 2024-03-28T15:04:54.128 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.129 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2024-03-28T15:04:54.130 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.130 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/7 2024-03-28T15:04:54.130 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.132 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2024-03-28T15:04:54.132 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.133 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/8 2024-03-28T15:04:54.133 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.134 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2024-03-28T15:04:54.135 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.135 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/9 2024-03-28T15:04:54.135 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.137 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2024-03-28T15:04:54.137 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.138 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/10 2024-03-28T15:04:54.138 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.139 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2024-03-28T15:04:54.140 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.140 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/11 2024-03-28T15:04:54.140 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.142 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2024-03-28T15:04:54.143 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.143 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/12 2024-03-28T15:04:54.143 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.144 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2024-03-28T15:04:54.145 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.145 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/13 2024-03-28T15:04:54.145 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.147 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2024-03-28T15:04:54.147 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.148 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/14 2024-03-28T15:04:54.148 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.149 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2024-03-28T15:04:54.150 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.150 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/15 2024-03-28T15:04:54.150 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.152 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2024-03-28T15:04:54.153 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.153 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/16 2024-03-28T15:04:54.153 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.154 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2024-03-28T15:04:54.155 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.155 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/17 2024-03-28T15:04:54.155 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.157 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2024-03-28T15:04:54.157 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.157 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/18 2024-03-28T15:04:54.157 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.159 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2024-03-28T15:04:54.160 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.160 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/19 2024-03-28T15:04:54.160 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.162 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2024-03-28T15:04:54.162 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.162 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/20 2024-03-28T15:04:54.162 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.164 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2024-03-28T15:04:54.165 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.165 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/21 2024-03-28T15:04:54.165 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.167 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2024-03-28T15:04:54.167 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.167 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/22 2024-03-28T15:04:54.167 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.169 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2024-03-28T15:04:54.170 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.170 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/23 2024-03-28T15:04:54.170 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.171 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2024-03-28T15:04:54.172 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.172 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/24 2024-03-28T15:04:54.172 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.174 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2024-03-28T15:04:54.175 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.175 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/25 2024-03-28T15:04:54.175 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.176 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2024-03-28T15:04:54.177 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.177 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/26 2024-03-28T15:04:54.177 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.179 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2024-03-28T15:04:54.179 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.179 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/27 2024-03-28T15:04:54.179 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.181 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2024-03-28T15:04:54.182 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.182 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/28 2024-03-28T15:04:54.182 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.184 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2024-03-28T15:04:54.184 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.184 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/29 2024-03-28T15:04:54.184 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.186 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2024-03-28T15:04:54.187 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.187 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/30 2024-03-28T15:04:54.187 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.189 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2024-03-28T15:04:54.189 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.189 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/31 2024-03-28T15:04:54.190 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.191 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2024-03-28T15:04:54.192 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.192 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/32 2024-03-28T15:04:54.192 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.194 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2024-03-28T15:04:54.194 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.194 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/33 2024-03-28T15:04:54.194 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.196 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2024-03-28T15:04:54.197 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.197 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/34 2024-03-28T15:04:54.197 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.199 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2024-03-28T15:04:54.199 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.199 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/35 2024-03-28T15:04:54.199 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.202 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2024-03-28T15:04:54.202 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.202 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/36 2024-03-28T15:04:54.202 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.204 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2024-03-28T15:04:54.204 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.205 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/37 2024-03-28T15:04:54.205 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.206 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2024-03-28T15:04:54.207 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.207 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/38 2024-03-28T15:04:54.207 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.209 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2024-03-28T15:04:54.209 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.209 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/39 2024-03-28T15:04:54.209 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.211 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2024-03-28T15:04:54.212 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.212 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/40 2024-03-28T15:04:54.212 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.214 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2024-03-28T15:04:54.214 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.214 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/41 2024-03-28T15:04:54.214 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.216 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2024-03-28T15:04:54.217 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.217 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/42 2024-03-28T15:04:54.217 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.222 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2024-03-28T15:04:54.223 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.223 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/43 2024-03-28T15:04:54.223 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.223 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2024-03-28T15:04:54.223 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.223 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/44 2024-03-28T15:04:54.224 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.226 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2024-03-28T15:04:54.226 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.227 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/45 2024-03-28T15:04:54.227 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.232 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2024-03-28T15:04:54.233 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.233 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/46 2024-03-28T15:04:54.233 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.240 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2024-03-28T15:04:54.241 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.241 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/47 2024-03-28T15:04:54.241 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.247 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2024-03-28T15:04:54.248 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.248 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/48 2024-03-28T15:04:54.248 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.253 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2024-03-28T15:04:54.254 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.254 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/49 2024-03-28T15:04:54.255 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.258 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2024-03-28T15:04:54.258 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.258 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/50 2024-03-28T15:04:54.258 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.262 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2024-03-28T15:04:54.263 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.263 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/51 2024-03-28T15:04:54.263 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.266 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2024-03-28T15:04:54.266 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.266 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/52 2024-03-28T15:04:54.266 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.270 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2024-03-28T15:04:54.270 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.270 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/53 2024-03-28T15:04:54.271 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.273 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2024-03-28T15:04:54.274 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.274 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/54 2024-03-28T15:04:54.274 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.276 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2024-03-28T15:04:54.277 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.277 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/55 2024-03-28T15:04:54.277 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.280 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2024-03-28T15:04:54.280 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.280 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/56 2024-03-28T15:04:54.281 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.282 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2024-03-28T15:04:54.283 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.283 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/57 2024-03-28T15:04:54.283 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.285 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2024-03-28T15:04:54.286 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.286 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/58 2024-03-28T15:04:54.286 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.288 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2024-03-28T15:04:54.289 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.289 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/59 2024-03-28T15:04:54.289 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.291 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2024-03-28T15:04:54.291 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.291 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/60 2024-03-28T15:04:54.291 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.293 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2024-03-28T15:04:54.294 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.294 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/61 2024-03-28T15:04:54.294 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.296 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2024-03-28T15:04:54.297 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.297 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/62 2024-03-28T15:04:54.297 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.299 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2024-03-28T15:04:54.300 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.300 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/63 2024-03-28T15:04:54.300 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.302 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2024-03-28T15:04:54.302 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.302 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/64 2024-03-28T15:04:54.303 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.305 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2024-03-28T15:04:54.305 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.306 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/65 2024-03-28T15:04:54.306 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.307 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2024-03-28T15:04:54.308 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.308 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/66 2024-03-28T15:04:54.308 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.310 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2024-03-28T15:04:54.311 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.311 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/67 2024-03-28T15:04:54.311 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.313 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2024-03-28T15:04:54.313 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.314 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/68 2024-03-28T15:04:54.314 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.316 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2024-03-28T15:04:54.316 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.316 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/69 2024-03-28T15:04:54.317 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.319 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2024-03-28T15:04:54.319 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.319 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/70 2024-03-28T15:04:54.319 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.322 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2024-03-28T15:04:54.322 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.322 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/71 2024-03-28T15:04:54.322 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.324 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2024-03-28T15:04:54.325 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.325 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/72 2024-03-28T15:04:54.325 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.327 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2024-03-28T15:04:54.328 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.328 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/73 2024-03-28T15:04:54.328 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.330 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2024-03-28T15:04:54.331 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.331 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/74 2024-03-28T15:04:54.331 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.333 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2024-03-28T15:04:54.333 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.334 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/75 2024-03-28T15:04:54.334 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.336 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2024-03-28T15:04:54.336 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.336 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/76 2024-03-28T15:04:54.337 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.341 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2024-03-28T15:04:54.342 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.342 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/77 2024-03-28T15:04:54.342 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.346 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2024-03-28T15:04:54.346 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.346 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/78 2024-03-28T15:04:54.346 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.350 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2024-03-28T15:04:54.351 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.351 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/79 2024-03-28T15:04:54.351 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.354 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2024-03-28T15:04:54.355 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.355 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/80 2024-03-28T15:04:54.355 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.358 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2024-03-28T15:04:54.359 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.359 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/81 2024-03-28T15:04:54.359 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.362 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2024-03-28T15:04:54.362 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.362 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/82 2024-03-28T15:04:54.363 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.365 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2024-03-28T15:04:54.365 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.365 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/83 2024-03-28T15:04:54.365 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.367 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2024-03-28T15:04:54.367 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.368 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/84 2024-03-28T15:04:54.368 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.369 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2024-03-28T15:04:54.370 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.370 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/85 2024-03-28T15:04:54.370 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.372 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2024-03-28T15:04:54.372 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.373 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/86 2024-03-28T15:04:54.373 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.374 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2024-03-28T15:04:54.375 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.375 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/87 2024-03-28T15:04:54.375 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.377 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2024-03-28T15:04:54.377 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.377 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/88 2024-03-28T15:04:54.378 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.379 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2024-03-28T15:04:54.380 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.380 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/89 2024-03-28T15:04:54.380 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.382 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2024-03-28T15:04:54.382 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.382 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/90 2024-03-28T15:04:54.382 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.384 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2024-03-28T15:04:54.385 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.385 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/91 2024-03-28T15:04:54.385 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.387 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2024-03-28T15:04:54.387 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.387 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/92 2024-03-28T15:04:54.387 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.389 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2024-03-28T15:04:54.390 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.390 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/93 2024-03-28T15:04:54.390 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.392 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2024-03-28T15:04:54.392 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.392 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/94 2024-03-28T15:04:54.392 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.394 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2024-03-28T15:04:54.395 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.395 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/95 2024-03-28T15:04:54.395 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.397 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2024-03-28T15:04:54.397 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.398 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/96 2024-03-28T15:04:54.398 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.399 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2024-03-28T15:04:54.400 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.400 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/97 2024-03-28T15:04:54.400 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.402 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2024-03-28T15:04:54.402 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.403 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/98 2024-03-28T15:04:54.403 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.405 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2024-03-28T15:04:54.405 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.405 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/99 2024-03-28T15:04:54.405 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.407 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2024-03-28T15:04:54.407 INFO:tasks.workunit.client.0.smithi186.stderr:+ for f in `seq 1 100` 2024-03-28T15:04:54.408 INFO:tasks.workunit.client.0.smithi186.stderr:+ stat .snap/foo/100 2024-03-28T15:04:54.408 INFO:tasks.workunit.client.0.smithi186.stderr:+ grep 'No such file' 2024-03-28T15:04:54.409 INFO:tasks.workunit.client.0.smithi186.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2024-03-28T15:04:54.410 INFO:tasks.workunit.client.0.smithi186.stderr:+ rmdir .snap/foo 2024-03-28T15:04:54.427 INFO:tasks.workunit.client.0.smithi186.stdout:OK 2024-03-28T15:04:54.428 INFO:tasks.workunit.client.0.smithi186.stderr:+ echo OK 2024-03-28T15:04:54.429 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T15:04:54.429 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:04:54.501 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2024-03-28T15:04:54.501 DEBUG:teuthology.orchestra.run.smithi186:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2024-03-28T15:04:54.565 INFO:tasks.workunit.client.0.smithi186.stderr:+ set -e 2024-03-28T15:04:54.565 INFO:tasks.workunit.client.0.smithi186.stderr:+ sudo git config --global http.sslVerify false 2024-03-28T15:04:54.582 INFO:tasks.workunit.client.0.smithi186.stderr:+ sudo git config --global http.postBuffer 1048576000 2024-03-28T15:04:54.599 INFO:tasks.workunit.client.0.smithi186.stderr:+ retried=false 2024-03-28T15:04:54.599 INFO:tasks.workunit.client.0.smithi186.stderr:+ trap -- retry EXIT 2024-03-28T15:04:54.599 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm -rf ceph 2024-03-28T15:04:54.599 INFO:tasks.workunit.client.0.smithi186.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2024-03-28T15:04:54.603 INFO:tasks.workunit.client.0.smithi186.stderr:Cloning into 'ceph'... 2024-03-28T15:04:55.862 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:04:55.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:05:26.024 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:05:26.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:05:47.511 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:05:47.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.622+0000 7fb0d2f25640 1 Processor -- start 2024-03-28T15:05:47.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.622+0000 7fb0d2f25640 1 -- start start 2024-03-28T15:05:47.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.623+0000 7fb0d2f25640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb0cc151990 0x7fb0cc0c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.623+0000 7fb0d2f25640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb0cc0b1e30 0x7fb0cc0c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.623+0000 7fb0d2f25640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0cc06caa0 con 0x7fb0cc0c2d00 2024-03-28T15:05:47.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.623+0000 7fb0d2f25640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb0cc0c1da0 con 0x7fb0cc151990 2024-03-28T15:05:47.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.623+0000 7fb0d2f25640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fb0cc06cd80 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.623+0000 7fb0d0cc1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb0cc151990 0x7fb0cc0c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.623+0000 7fb0d0cc1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb0cc151990 0x7fb0cc0c2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40614/0 (socket says 172.21.15.186:40614) 2024-03-28T15:05:47.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.623+0000 7fb0d0cc1640 1 -- 172.21.15.186:0/803137804 learned_addr learned my addr 172.21.15.186:0/803137804 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:05:47.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.623+0000 7fb0cbfff640 1 --2- 172.21.15.186:0/803137804 >> v2:172.21.15.191:3300/0 conn(0x7fb0cc0b1e30 0x7fb0cc0c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.623+0000 7fb0d0cc1640 1 -- 172.21.15.186:0/803137804 >> v2:172.21.15.191:3300/0 conn(0x7fb0cc0b1e30 msgr2=0x7fb0cc0c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:05:47.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.623+0000 7fb0d0cc1640 1 --2- 172.21.15.186:0/803137804 >> v2:172.21.15.191:3300/0 conn(0x7fb0cc0b1e30 0x7fb0cc0c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.624+0000 7fb0d0cc1640 1 -- 172.21.15.186:0/803137804 >> v1:172.21.15.191:6789/0 conn(0x7fb0cc0c2d00 legacy=0x7fb0cc0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:47.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.624+0000 7fb0d0cc1640 1 -- 172.21.15.186:0/803137804 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0cc0c1340 con 0x7fb0cc151990 2024-03-28T15:05:47.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.624+0000 7fb0d0cc1640 1 --2- 172.21.15.186:0/803137804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc151990 0x7fb0cc0c2230 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0724b0 tx=0x7fb0bc02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2719a5456cfd01ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:47.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.624+0000 7fb0cb7fe640 1 -- 172.21.15.186:0/803137804 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc02f850 con 0x7fb0cc151990 2024-03-28T15:05:47.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.624+0000 7fb0cb7fe640 1 -- 172.21.15.186:0/803137804 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0bc02f9f0 con 0x7fb0cc151990 2024-03-28T15:05:47.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.624+0000 7fb0cb7fe640 1 -- 172.21.15.186:0/803137804 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc02fdc0 con 0x7fb0cc151990 2024-03-28T15:05:47.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.625+0000 7fb0d2f25640 1 -- 172.21.15.186:0/803137804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc151990 msgr2=0x7fb0cc0c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:47.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.625+0000 7fb0d2f25640 1 --2- 172.21.15.186:0/803137804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc151990 0x7fb0cc0c2230 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0724b0 tx=0x7fb0bc02ba30 comp rx=0 tx=0).stop 2024-03-28T15:05:47.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.625+0000 7fb0d2f25640 1 -- 172.21.15.186:0/803137804 shutdown_connections 2024-03-28T15:05:47.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.625+0000 7fb0d2f25640 1 --2- 172.21.15.186:0/803137804 >> v2:172.21.15.191:3300/0 conn(0x7fb0cc0b1e30 0x7fb0cc0c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.625+0000 7fb0d2f25640 1 --2- 172.21.15.186:0/803137804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc151990 0x7fb0cc0c2230 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.625+0000 7fb0d2f25640 1 -- 172.21.15.186:0/803137804 >> 172.21.15.186:0/803137804 conn(0x7fb0cc0c8a20 msgr2=0x7fb0cc0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:47.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.625+0000 7fb0d2f25640 1 -- 172.21.15.186:0/803137804 shutdown_connections 2024-03-28T15:05:47.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.625+0000 7fb0d2f25640 1 -- 172.21.15.186:0/803137804 wait complete. 2024-03-28T15:05:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.626+0000 7fb0d2f25640 1 Processor -- start 2024-03-28T15:05:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.626+0000 7fb0d2f25640 1 -- start start 2024-03-28T15:05:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.627+0000 7fb0d2f25640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0cc0b1e30 0x7fb0cc14d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.627+0000 7fb0d2f25640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0cc0c2d00 0x7fb0cc0c3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.627+0000 7fb0d2f25640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc151990 0x7fb0cc1f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.627+0000 7fb0d2f25640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc073dd0 con 0x7fb0cc151990 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.627+0000 7fb0d2f25640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc0c1900 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.627+0000 7fb0d2f25640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb0cc073f50 con 0x7fb0cc0c2d00 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.627+0000 7fb0d14c2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc151990 0x7fb0cc1f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.627+0000 7fb0d14c2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc151990 0x7fb0cc1f0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40622/0 (socket says 172.21.15.186:40622) 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.627+0000 7fb0d14c2640 1 -- 172.21.15.186:0/1818430965 learned_addr learned my addr 172.21.15.186:0/1818430965 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.627+0000 7fb0d0cc1640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0cc0b1e30 0x7fb0cc14d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.627+0000 7fb0cbfff640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0cc0c2d00 0x7fb0cc0c3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0d0cc1640 1 -- 172.21.15.186:0/1818430965 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0cc0c2d00 msgr2=0x7fb0cc0c3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0d0cc1640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0cc0c2d00 0x7fb0cc0c3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0d0cc1640 1 -- 172.21.15.186:0/1818430965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc151990 msgr2=0x7fb0cc1f0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0d0cc1640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc151990 0x7fb0cc1f0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0d0cc1640 1 -- 172.21.15.186:0/1818430965 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0cc1f0730 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0d14c2640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc151990 0x7fb0cc1f0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:05:47.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0cbfff640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0cc0c2d00 0x7fb0cc0c3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0d0cc1640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0cc0b1e30 0x7fb0cc14d5d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc06e690 tx=0x7fb0bc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0c97fa640 1 -- 172.21.15.186:0/1818430965 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc047070 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0c97fa640 1 -- 172.21.15.186:0/1818430965 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0bc037be0 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0d2f25640 1 -- 172.21.15.186:0/1818430965 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0cc0cb2f0 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0c97fa640 1 -- 172.21.15.186:0/1818430965 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc042480 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.628+0000 7fb0d2f25640 1 -- 172.21.15.186:0/1818430965 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0cc0cb890 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.629+0000 7fb0d2f25640 1 -- 172.21.15.186:0/1818430965 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0cc069500 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.629+0000 7fb0c97fa640 1 -- 172.21.15.186:0/1818430965 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb0bc04a070 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.630+0000 7fb0c97fa640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0b806c530 0x7fb0b806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.630+0000 7fb0c97fa640 1 -- 172.21.15.186:0/1818430965 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7fb0bc0ad930 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.630+0000 7fb0cbfff640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0b806c530 0x7fb0b806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.630+0000 7fb0cbfff640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0b806c530 0x7fb0b806e9f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc1e4e00 tx=0x7fb0c0009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:47.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.635+0000 7fb0c97fa640 1 -- 172.21.15.186:0/1818430965 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb0cc1f0730 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.848+0000 7fb0d2f25640 1 -- 172.21.15.186:0/1818430965 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0cc0b1a00 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.849+0000 7fb0c97fa640 1 -- 172.21.15.186:0/1818430965 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6848 (secure 0 0 0) 0x7fb0bc033030 con 0x7fb0cc0b1e30 2024-03-28T15:05:47.849 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:05:47.850 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:05:47.850 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 19 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 -- 172.21.15.186:0/1818430965 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0b806c530 msgr2=0x7fb0b806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0b806c530 0x7fb0b806e9f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc1e4e00 tx=0x7fb0c0009190 comp rx=0 tx=0).stop 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 -- 172.21.15.186:0/1818430965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0cc0b1e30 msgr2=0x7fb0cc14d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0cc0b1e30 0x7fb0cc14d5d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc06e690 tx=0x7fb0bc004730 comp rx=0 tx=0).stop 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 -- 172.21.15.186:0/1818430965 shutdown_connections 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0b806c530 0x7fb0b806e9f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc151990 0x7fb0cc1f0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0cc0c2d00 0x7fb0cc0c3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 --2- 172.21.15.186:0/1818430965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0cc0b1e30 0x7fb0cc14d5d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 -- 172.21.15.186:0/1818430965 >> 172.21.15.186:0/1818430965 conn(0x7fb0cc0c8a20 msgr2=0x7fb0cc06e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 -- 172.21.15.186:0/1818430965 shutdown_connections 2024-03-28T15:05:47.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.851+0000 7fb0d2f25640 1 -- 172.21.15.186:0/1818430965 wait complete. 2024-03-28T15:05:47.862 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2024-03-28T15:05:47.863 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-03-28T15:05:47.863 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:05:47.875 INFO:tasks.ceph.mds.f:Stopped 2024-03-28T15:05:47.876 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:05:47.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.976+0000 7fb462bd6640 1 Processor -- start 2024-03-28T15:05:47.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.977+0000 7fb462bd6640 1 -- start start 2024-03-28T15:05:47.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.977+0000 7fb462bd6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb45c151990 0x7fb45c0c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.977+0000 7fb462bd6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb45c0b1e30 0x7fb45c0c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.977+0000 7fb462bd6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb45c06caa0 con 0x7fb45c0c2d00 2024-03-28T15:05:47.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.977+0000 7fb462bd6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fb45c0c1da0 con 0x7fb45c151990 2024-03-28T15:05:47.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.977+0000 7fb462bd6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb45c06cd80 con 0x7fb45c0b1e30 2024-03-28T15:05:47.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.978+0000 7fb461173640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fb45c0c2d00 0x7fb45c0712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:50044/0 (socket says 172.21.15.186:50044) 2024-03-28T15:05:47.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.978+0000 7fb460972640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb45c151990 0x7fb45c0c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.978+0000 7fb461173640 1 -- 172.21.15.186:0/3157111706 learned_addr learned my addr 172.21.15.186:0/3157111706 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:05:47.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.978+0000 7fb45bfff640 1 --2- 172.21.15.186:0/3157111706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb45c0b1e30 0x7fb45c0c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.978+0000 7fb45b7fe640 1 -- 172.21.15.186:0/3157111706 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1819930753 0 0) 0x7fb45c06caa0 con 0x7fb45c0c2d00 2024-03-28T15:05:47.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.978+0000 7fb45b7fe640 1 -- 172.21.15.186:0/3157111706 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb44c003520 con 0x7fb45c0c2d00 2024-03-28T15:05:47.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.978+0000 7fb460972640 1 -- 172.21.15.186:0/3157111706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb45c0b1e30 msgr2=0x7fb45c0c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:05:47.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.978+0000 7fb460972640 1 --2- 172.21.15.186:0/3157111706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb45c0b1e30 0x7fb45c0c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.978+0000 7fb460972640 1 -- 172.21.15.186:0/3157111706 >> v1:172.21.15.191:6789/0 conn(0x7fb45c0c2d00 legacy=0x7fb45c0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:47.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.978+0000 7fb460972640 1 -- 172.21.15.186:0/3157111706 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45c0c1340 con 0x7fb45c151990 2024-03-28T15:05:47.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.978+0000 7fb45bfff640 1 --2- 172.21.15.186:0/3157111706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb45c0b1e30 0x7fb45c0c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:05:47.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb460972640 1 --2- 172.21.15.186:0/3157111706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb45c151990 0x7fb45c0c2230 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb4500097a0 tx=0x7fb45002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddd30f069fb6610d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:47.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb45b7fe640 1 -- 172.21.15.186:0/3157111706 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb450034070 con 0x7fb45c151990 2024-03-28T15:05:47.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb45b7fe640 1 -- 172.21.15.186:0/3157111706 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb450030610 con 0x7fb45c151990 2024-03-28T15:05:47.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb45b7fe640 1 -- 172.21.15.186:0/3157111706 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4500309c0 con 0x7fb45c151990 2024-03-28T15:05:47.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb462bd6640 1 -- 172.21.15.186:0/3157111706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb45c151990 msgr2=0x7fb45c0c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:47.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb462bd6640 1 --2- 172.21.15.186:0/3157111706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb45c151990 0x7fb45c0c2230 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb4500097a0 tx=0x7fb45002b960 comp rx=0 tx=0).stop 2024-03-28T15:05:47.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb462bd6640 1 -- 172.21.15.186:0/3157111706 shutdown_connections 2024-03-28T15:05:47.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb462bd6640 1 --2- 172.21.15.186:0/3157111706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb45c0b1e30 0x7fb45c0c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb462bd6640 1 --2- 172.21.15.186:0/3157111706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb45c151990 0x7fb45c0c2230 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb462bd6640 1 -- 172.21.15.186:0/3157111706 >> 172.21.15.186:0/3157111706 conn(0x7fb45c0c8a20 msgr2=0x7fb45c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:47.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb462bd6640 1 -- 172.21.15.186:0/3157111706 shutdown_connections 2024-03-28T15:05:47.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.979+0000 7fb462bd6640 1 -- 172.21.15.186:0/3157111706 wait complete. 2024-03-28T15:05:47.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.981+0000 7fb462bd6640 1 Processor -- start 2024-03-28T15:05:47.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.981+0000 7fb462bd6640 1 -- start start 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.981+0000 7fb462bd6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb45c0b1e30 0x7fb45c14a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.981+0000 7fb462bd6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb45c0c2d00 0x7fb45c1e0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.981+0000 7fb462bd6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c151990 0x7fb45c1e0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.981+0000 7fb462bd6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c073dd0 con 0x7fb45c151990 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.981+0000 7fb462bd6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c0c1900 con 0x7fb45c0b1e30 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.981+0000 7fb462bd6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb45c073f50 con 0x7fb45c0c2d00 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb461173640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c151990 0x7fb45c1e0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb460972640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb45c0b1e30 0x7fb45c14a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb460972640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb45c0b1e30 0x7fb45c14a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:57540/0 (socket says 172.21.15.186:57540) 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb45bfff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb45c0c2d00 0x7fb45c1e0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb460972640 1 -- 172.21.15.186:0/1135167934 learned_addr learned my addr 172.21.15.186:0/1135167934 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:05:47.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb461173640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c151990 0x7fb45c1e0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40624/0 (socket says 172.21.15.186:40624) 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb460972640 1 -- 172.21.15.186:0/1135167934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb45c0c2d00 msgr2=0x7fb45c1e0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb460972640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb45c0c2d00 0x7fb45c1e0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb460972640 1 -- 172.21.15.186:0/1135167934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c151990 msgr2=0x7fb45c1e0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb460972640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c151990 0x7fb45c1e0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb460972640 1 -- 172.21.15.186:0/1135167934 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45c1f07d0 con 0x7fb45c0b1e30 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb460972640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb45c0b1e30 0x7fb45c14a150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb45002bf70 tx=0x7fb4500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb4597fa640 1 -- 172.21.15.186:0/1135167934 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb450034040 con 0x7fb45c0b1e30 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb4597fa640 1 -- 172.21.15.186:0/1135167934 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb45003aaa0 con 0x7fb45c0b1e30 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb462bd6640 1 -- 172.21.15.186:0/1135167934 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45c1f0ab0 con 0x7fb45c0b1e30 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb4597fa640 1 -- 172.21.15.186:0/1135167934 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb450006e10 con 0x7fb45c0b1e30 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.982+0000 7fb462bd6640 1 -- 172.21.15.186:0/1135167934 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb45c1f1070 con 0x7fb45c0b1e30 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.983+0000 7fb462bd6640 1 -- 172.21.15.186:0/1135167934 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb45c069500 con 0x7fb45c0b1e30 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.984+0000 7fb4597fa640 1 -- 172.21.15.186:0/1135167934 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb450037070 con 0x7fb45c0b1e30 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.984+0000 7fb4597fa640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb44006c530 0x7fb44006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:47.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.984+0000 7fb4597fa640 1 -- 172.21.15.186:0/1135167934 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7fb4500ad560 con 0x7fb45c0b1e30 2024-03-28T15:05:47.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.984+0000 7fb45bfff640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb44006c530 0x7fb44006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:47.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.985+0000 7fb45bfff640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb44006c530 0x7fb44006e9f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb448009560 tx=0x7fb448009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:47.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:47.989+0000 7fb4597fa640 1 -- 172.21.15.186:0/1135167934 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb45003f070 con 0x7fb45c0b1e30 2024-03-28T15:05:48.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.201+0000 7fb462bd6640 1 -- 172.21.15.186:0/1135167934 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb45c0b1a00 con 0x7fb45c0b1e30 2024-03-28T15:05:48.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.202+0000 7fb4597fa640 1 -- 172.21.15.186:0/1135167934 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6848 (secure 0 0 0) 0x7fb45007b370 con 0x7fb45c0b1e30 2024-03-28T15:05:48.202 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:05:48.203 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:05:48.203 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 19 2024-03-28T15:05:48.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 -- 172.21.15.186:0/1135167934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb44006c530 msgr2=0x7fb44006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:48.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb44006c530 0x7fb44006e9f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb448009560 tx=0x7fb448009190 comp rx=0 tx=0).stop 2024-03-28T15:05:48.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 -- 172.21.15.186:0/1135167934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb45c0b1e30 msgr2=0x7fb45c14a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:48.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb45c0b1e30 0x7fb45c14a150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb45002bf70 tx=0x7fb4500046c0 comp rx=0 tx=0).stop 2024-03-28T15:05:48.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 -- 172.21.15.186:0/1135167934 shutdown_connections 2024-03-28T15:05:48.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb44006c530 0x7fb44006e9f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:48.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c151990 0x7fb45c1e0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:48.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb45c0c2d00 0x7fb45c1e0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:48.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 --2- 172.21.15.186:0/1135167934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb45c0b1e30 0x7fb45c14a150 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:48.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 -- 172.21.15.186:0/1135167934 >> 172.21.15.186:0/1135167934 conn(0x7fb45c0c8a20 msgr2=0x7fb45c0c2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:48.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 -- 172.21.15.186:0/1135167934 shutdown_connections 2024-03-28T15:05:48.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:48.204+0000 7fb462bd6640 1 -- 172.21.15.186:0/1135167934 wait complete. 2024-03-28T15:05:48.215 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:05:50.217 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:05:50.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.330+0000 7fa10e3bc640 1 Processor -- start 2024-03-28T15:05:50.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.330+0000 7fa10e3bc640 1 -- start start 2024-03-28T15:05:50.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa10e3bc640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa1080b1e30 0x7fa1080712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:50.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa10e3bc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa1080c2d00 0x7fa1080717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:50.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa10e3bc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10806caa0 con 0x7fa108151990 2024-03-28T15:05:50.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa10e3bc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa1080c1da0 con 0x7fa1080c2d00 2024-03-28T15:05:50.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa10e3bc640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fa10806cd80 con 0x7fa1080b1e30 2024-03-28T15:05:50.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa10c959640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa1080c2d00 0x7fa1080717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:50.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa107fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fa108151990 0x7fa1080c2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:56570/0 (socket says 172.21.15.186:56570) 2024-03-28T15:05:50.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa107fff640 1 -- 172.21.15.186:0/3866139400 learned_addr learned my addr 172.21.15.186:0/3866139400 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:05:50.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa1077fe640 1 --2- 172.21.15.186:0/3866139400 >> v2:172.21.15.191:3300/0 conn(0x7fa1080b1e30 0x7fa1080712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:50.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa10c959640 1 -- 172.21.15.186:0/3866139400 >> v2:172.21.15.191:3300/0 conn(0x7fa1080b1e30 msgr2=0x7fa1080712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:05:50.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa10c959640 1 --2- 172.21.15.186:0/3866139400 >> v2:172.21.15.191:3300/0 conn(0x7fa1080b1e30 0x7fa1080712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:50.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa10c959640 1 -- 172.21.15.186:0/3866139400 >> v1:172.21.15.186:6789/0 conn(0x7fa108151990 legacy=0x7fa1080c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:50.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.332+0000 7fa10c959640 1 -- 172.21.15.186:0/3866139400 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1080c1340 con 0x7fa1080c2d00 2024-03-28T15:05:50.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.331+0000 7fa106ffd640 1 -- 172.21.15.186:0/3866139400 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3330813802 0 0) 0x7fa10806caa0 con 0x7fa108151990 2024-03-28T15:05:50.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.332+0000 7fa10c959640 1 --2- 172.21.15.186:0/3866139400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1080c2d00 0x7fa1080717f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa1080724b0 tx=0x7fa10002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948b8ca0594f2bcf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:50.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.332+0000 7fa106ffd640 1 -- 172.21.15.186:0/3866139400 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa100039070 con 0x7fa1080c2d00 2024-03-28T15:05:50.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.332+0000 7fa106ffd640 1 -- 172.21.15.186:0/3866139400 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa10002cd00 con 0x7fa1080c2d00 2024-03-28T15:05:50.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.332+0000 7fa106ffd640 1 -- 172.21.15.186:0/3866139400 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa100034660 con 0x7fa1080c2d00 2024-03-28T15:05:50.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.333+0000 7fa10e3bc640 1 -- 172.21.15.186:0/3866139400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1080c2d00 msgr2=0x7fa1080717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:50.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.333+0000 7fa10e3bc640 1 --2- 172.21.15.186:0/3866139400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1080c2d00 0x7fa1080717f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa1080724b0 tx=0x7fa10002bf50 comp rx=0 tx=0).stop 2024-03-28T15:05:50.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.333+0000 7fa10e3bc640 1 -- 172.21.15.186:0/3866139400 shutdown_connections 2024-03-28T15:05:50.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.333+0000 7fa10e3bc640 1 --2- 172.21.15.186:0/3866139400 >> v2:172.21.15.191:3300/0 conn(0x7fa1080b1e30 0x7fa1080712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:50.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.333+0000 7fa10e3bc640 1 --2- 172.21.15.186:0/3866139400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1080c2d00 0x7fa1080717f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:50.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.333+0000 7fa10e3bc640 1 -- 172.21.15.186:0/3866139400 >> 172.21.15.186:0/3866139400 conn(0x7fa1080c8a20 msgr2=0x7fa1080c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:50.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.333+0000 7fa10e3bc640 1 -- 172.21.15.186:0/3866139400 shutdown_connections 2024-03-28T15:05:50.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.333+0000 7fa10e3bc640 1 -- 172.21.15.186:0/3866139400 wait complete. 2024-03-28T15:05:50.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.334+0000 7fa10e3bc640 1 Processor -- start 2024-03-28T15:05:50.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.334+0000 7fa10e3bc640 1 -- start start 2024-03-28T15:05:50.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa10e3bc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1080b1e30 0x7fa1080c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:50.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa10e3bc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1080c2d00 0x7fa10813d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:50.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa10e3bc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa108151990 0x7fa10813def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:50.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa10e3bc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa108071d30 con 0x7fa1080c2d00 2024-03-28T15:05:50.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa10e3bc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa1080c1900 con 0x7fa108151990 2024-03-28T15:05:50.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa10e3bc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa1080c2940 con 0x7fa1080b1e30 2024-03-28T15:05:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa1077fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1080c2d00 0x7fa10813d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa1077fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1080c2d00 0x7fa10813d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52130/0 (socket says 172.21.15.186:52130) 2024-03-28T15:05:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa1077fe640 1 -- 172.21.15.186:0/1928928428 learned_addr learned my addr 172.21.15.186:0/1928928428 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:05:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa10c959640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa108151990 0x7fa10813def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa107fff640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1080b1e30 0x7fa1080c47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.335+0000 7fa1077fe640 1 -- 172.21.15.186:0/1928928428 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1080b1e30 msgr2=0x7fa1080c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.336+0000 7fa1077fe640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1080b1e30 0x7fa1080c47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.336+0000 7fa1077fe640 1 -- 172.21.15.186:0/1928928428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa108151990 msgr2=0x7fa10813def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.336+0000 7fa1077fe640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa108151990 0x7fa10813def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.336+0000 7fa1077fe640 1 -- 172.21.15.186:0/1928928428 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10813e430 con 0x7fa1080c2d00 2024-03-28T15:05:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.336+0000 7fa1077fe640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1080c2d00 0x7fa10813d9b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00d600 tx=0x7fa0fc00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.336+0000 7fa104ff9640 1 -- 172.21.15.186:0/1928928428 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc004240 con 0x7fa1080c2d00 2024-03-28T15:05:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.336+0000 7fa104ff9640 1 -- 172.21.15.186:0/1928928428 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0fc0043e0 con 0x7fa1080c2d00 2024-03-28T15:05:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.336+0000 7fa10e3bc640 1 -- 172.21.15.186:0/1928928428 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa10813c6d0 con 0x7fa1080c2d00 2024-03-28T15:05:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.336+0000 7fa104ff9640 1 -- 172.21.15.186:0/1928928428 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc003830 con 0x7fa1080c2d00 2024-03-28T15:05:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.336+0000 7fa10e3bc640 1 -- 172.21.15.186:0/1928928428 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1080ca8d0 con 0x7fa1080c2d00 2024-03-28T15:05:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.337+0000 7fa10e3bc640 1 -- 172.21.15.186:0/1928928428 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa108069500 con 0x7fa1080c2d00 2024-03-28T15:05:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.337+0000 7fa104ff9640 1 -- 172.21.15.186:0/1928928428 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa0fc0039d0 con 0x7fa1080c2d00 2024-03-28T15:05:50.343 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.338+0000 7fa104ff9640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa0f406c560 0x7fa0f406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:50.343 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.338+0000 7fa104ff9640 1 -- 172.21.15.186:0/1928928428 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7fa0fc08baf0 con 0x7fa1080c2d00 2024-03-28T15:05:50.343 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.338+0000 7fa107fff640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa0f406c560 0x7fa0f406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:50.343 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.338+0000 7fa107fff640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa0f406c560 0x7fa0f406ea20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8007fd0 tx=0x7fa0f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:50.343 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.343+0000 7fa104ff9640 1 -- 172.21.15.186:0/1928928428 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa10813c6d0 con 0x7fa1080c2d00 2024-03-28T15:05:50.554 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.554+0000 7fa10e3bc640 1 -- 172.21.15.186:0/1928928428 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1080b1a00 con 0x7fa1080c2d00 2024-03-28T15:05:50.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.554+0000 7fa104ff9640 1 -- 172.21.15.186:0/1928928428 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6848 (secure 0 0 0) 0x7fa0fc059fd0 con 0x7fa1080c2d00 2024-03-28T15:05:50.555 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:05:50.555 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:05:50.555 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 19 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 -- 172.21.15.186:0/1928928428 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa0f406c560 msgr2=0x7fa0f406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa0f406c560 0x7fa0f406ea20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8007fd0 tx=0x7fa0f8009040 comp rx=0 tx=0).stop 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 -- 172.21.15.186:0/1928928428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1080c2d00 msgr2=0x7fa10813d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1080c2d00 0x7fa10813d9b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00d600 tx=0x7fa0fc00dac0 comp rx=0 tx=0).stop 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 -- 172.21.15.186:0/1928928428 shutdown_connections 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa0f406c560 0x7fa0f406ea20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa108151990 0x7fa10813def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1080c2d00 0x7fa10813d9b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 --2- 172.21.15.186:0/1928928428 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1080b1e30 0x7fa1080c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 -- 172.21.15.186:0/1928928428 >> 172.21.15.186:0/1928928428 conn(0x7fa1080c8a20 msgr2=0x7fa1080c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 -- 172.21.15.186:0/1928928428 shutdown_connections 2024-03-28T15:05:50.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:50.556+0000 7fa10e3bc640 1 -- 172.21.15.186:0/1928928428 wait complete. 2024-03-28T15:05:50.568 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:05:52.570 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:05:52.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.681+0000 7f978a13e640 1 Processor -- start 2024-03-28T15:05:52.682 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.681+0000 7f978a13e640 1 -- start start 2024-03-28T15:05:52.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.682+0000 7f978a13e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841512b0 0x7f97840c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:52.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.682+0000 7f978a13e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97840c1b40 con 0x7f97840733f0 2024-03-28T15:05:52.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.682+0000 7f978a13e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97840c15c0 con 0x7f9784150ed0 2024-03-28T15:05:52.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.682+0000 7f978a13e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f97840c1e20 con 0x7f97841512b0 2024-03-28T15:05:52.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.682+0000 7f9783fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f97840733f0 0x7f97840b1750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:56584/0 (socket says 172.21.15.186:56584) 2024-03-28T15:05:52.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.682+0000 7f9783fff640 1 -- 172.21.15.186:0/3640143724 learned_addr learned my addr 172.21.15.186:0/3640143724 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:05:52.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.682+0000 7f9782ffd640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841512b0 0x7f97840c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:52.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.682+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4029964734 0 0) 0x7f97840c1b40 con 0x7f97840733f0 2024-03-28T15:05:52.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.682+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f976c003520 con 0x7f97840733f0 2024-03-28T15:05:52.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649582430 0 0) 0x7f97840c15c0 con 0x7f9784150ed0 2024-03-28T15:05:52.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97840c1b40 con 0x7f9784150ed0 2024-03-28T15:05:52.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3368415580 0 0) 0x7f976c003520 con 0x7f97840733f0 2024-03-28T15:05:52.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841512b0 msgr2=0x7f97840c0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:05:52.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841512b0 0x7f97840c0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:52.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 >> v1:172.21.15.191:6789/0 conn(0x7f9784150ed0 legacy=0x7f9784156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:52.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9784072bf0 con 0x7f97840733f0 2024-03-28T15:05:52.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f977c0019f0 con 0x7f97840733f0 2024-03-28T15:05:52.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f9782ffd640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841512b0 0x7f97840c0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:05:52.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 >> v1:172.21.15.186:6789/0 conn(0x7f97840733f0 legacy=0x7f97840b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:52.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841512b0 0x7f976c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:52.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f976c004100 0x7f976c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:52.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f976c004d40 0x7f976c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:52.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97840c1e20 con 0x7f976c004100 2024-03-28T15:05:52.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f976c00bad0 con 0x7f976c004d40 2024-03-28T15:05:52.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f976c00bcd0 con 0x7f97841512b0 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f9782ffd640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f976c004100 0x7f976c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f97837fe640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841512b0 0x7f976c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f9783fff640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f976c004d40 0x7f976c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f9782ffd640 1 -- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841512b0 msgr2=0x7f976c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f9782ffd640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841512b0 0x7f976c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f9782ffd640 1 -- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f976c004d40 msgr2=0x7f976c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f9782ffd640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f976c004d40 0x7f976c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.683+0000 7f9782ffd640 1 -- 172.21.15.186:0/3640143724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976c003520 con 0x7f976c004100 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.684+0000 7f9782ffd640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f976c004100 0x7f976c004800 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9784151690 tx=0x7f976c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f68c83992c2f41 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.684+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778039070 con 0x7f976c004100 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.684+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f977802cce0 con 0x7f976c004100 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.684+0000 7f97827fc640 1 -- 172.21.15.186:0/3640143724 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977800b040 con 0x7f976c004100 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.685+0000 7f978a13e640 1 -- 172.21.15.186:0/3640143724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f976c004100 msgr2=0x7f976c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:52.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.685+0000 7f978a13e640 1 --2- 172.21.15.186:0/3640143724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f976c004100 0x7f976c004800 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9784151690 tx=0x7f976c005830 comp rx=0 tx=0).stop 2024-03-28T15:05:52.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.685+0000 7f9783fff640 1 -- 172.21.15.186:0/3640143724 reap_dead start 2024-03-28T15:05:52.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.685+0000 7f978a13e640 1 -- 172.21.15.186:0/3640143724 shutdown_connections 2024-03-28T15:05:52.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.685+0000 7f978a13e640 1 -- 172.21.15.186:0/3640143724 >> 172.21.15.186:0/3640143724 conn(0x7f97840c8a20 msgr2=0x7f97840c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:52.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.685+0000 7f978a13e640 1 -- 172.21.15.186:0/3640143724 shutdown_connections 2024-03-28T15:05:52.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.685+0000 7f978a13e640 1 -- 172.21.15.186:0/3640143724 wait complete. 2024-03-28T15:05:52.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.686+0000 7f978a13e640 1 Processor -- start 2024-03-28T15:05:52.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.686+0000 7f978a13e640 1 -- start start 2024-03-28T15:05:52.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f978a13e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f976c004100 0x7f97840c1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:52.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f978a13e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97841e0720 0x7f97840cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:52.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f978a13e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841df100 0x7f97841df560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:52.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f978a13e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9784072f60 con 0x7f97841e0720 2024-03-28T15:05:52.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f978a13e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f97840c18a0 con 0x7f976c004100 2024-03-28T15:05:52.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f978a13e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f97840b1e60 con 0x7f97841df100 2024-03-28T15:05:52.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f9782ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97841e0720 0x7f97840cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:52.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f9782ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97841e0720 0x7f97840cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52136/0 (socket says 172.21.15.186:52136) 2024-03-28T15:05:52.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f9782ffd640 1 -- 172.21.15.186:0/734405568 learned_addr learned my addr 172.21.15.186:0/734405568 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f97837fe640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f976c004100 0x7f97840c1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f9783fff640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841df100 0x7f97841df560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f9782ffd640 1 -- 172.21.15.186:0/734405568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841df100 msgr2=0x7f97841df560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f9782ffd640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841df100 0x7f97841df560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f9782ffd640 1 -- 172.21.15.186:0/734405568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f976c004100 msgr2=0x7f97840c1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f9782ffd640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f976c004100 0x7f97840c1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f9782ffd640 1 -- 172.21.15.186:0/734405568 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97840c7990 con 0x7f97841e0720 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f97837fe640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f976c004100 0x7f97840c1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f9783fff640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841df100 0x7f97841df560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f9782ffd640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97841e0720 0x7f97840cb110 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f977802c690 tx=0x7f9778004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.687+0000 7f976bfff640 1 -- 172.21.15.186:0/734405568 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778042020 con 0x7f97841e0720 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.688+0000 7f976bfff640 1 -- 172.21.15.186:0/734405568 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f977803a040 con 0x7f97841e0720 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.688+0000 7f978a13e640 1 -- 172.21.15.186:0/734405568 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97841e0e60 con 0x7f97841e0720 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.688+0000 7f976bfff640 1 -- 172.21.15.186:0/734405568 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778039070 con 0x7f97841e0720 2024-03-28T15:05:52.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.688+0000 7f978a13e640 1 -- 172.21.15.186:0/734405568 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97841eca40 con 0x7f97841e0720 2024-03-28T15:05:52.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.688+0000 7f978a13e640 1 -- 172.21.15.186:0/734405568 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9784069500 con 0x7f97841e0720 2024-03-28T15:05:52.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.689+0000 7f976bfff640 1 -- 172.21.15.186:0/734405568 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9778045020 con 0x7f97841e0720 2024-03-28T15:05:52.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.689+0000 7f976bfff640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f977006c530 0x7f977006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:52.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.689+0000 7f976bfff640 1 -- 172.21.15.186:0/734405568 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7f97780ad900 con 0x7f97841e0720 2024-03-28T15:05:52.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.689+0000 7f97837fe640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f977006c530 0x7f977006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:52.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.690+0000 7f97837fe640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f977006c530 0x7f977006e9f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f97740055e0 tx=0x7f9774009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:52.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.694+0000 7f976bfff640 1 -- 172.21.15.186:0/734405568 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f977804c020 con 0x7f97841e0720 2024-03-28T15:05:52.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.908+0000 7f978a13e640 1 -- 172.21.15.186:0/734405568 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97840c81e0 con 0x7f97841e0720 2024-03-28T15:05:52.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.909+0000 7f976bfff640 1 -- 172.21.15.186:0/734405568 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6848 (secure 0 0 0) 0x7f97840c7990 con 0x7f97841e0720 2024-03-28T15:05:52.909 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:05:52.909 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:05:52.909 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 19 2024-03-28T15:05:52.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.910+0000 7f978a13e640 1 -- 172.21.15.186:0/734405568 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f977006c530 msgr2=0x7f977006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:52.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.910+0000 7f978a13e640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f977006c530 0x7f977006e9f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f97740055e0 tx=0x7f9774009040 comp rx=0 tx=0).stop 2024-03-28T15:05:52.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.911+0000 7f978a13e640 1 -- 172.21.15.186:0/734405568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97841e0720 msgr2=0x7f97840cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:52.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.911+0000 7f978a13e640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97841e0720 0x7f97840cb110 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f977802c690 tx=0x7f9778004730 comp rx=0 tx=0).stop 2024-03-28T15:05:52.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.911+0000 7f978a13e640 1 -- 172.21.15.186:0/734405568 shutdown_connections 2024-03-28T15:05:52.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.911+0000 7f978a13e640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f977006c530 0x7f977006e9f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:52.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.911+0000 7f978a13e640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97841df100 0x7f97841df560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:52.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.911+0000 7f978a13e640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97841e0720 0x7f97840cb110 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:52.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.911+0000 7f978a13e640 1 --2- 172.21.15.186:0/734405568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f976c004100 0x7f97840c1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:52.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.911+0000 7f978a13e640 1 -- 172.21.15.186:0/734405568 >> 172.21.15.186:0/734405568 conn(0x7f97840c8a20 msgr2=0x7f978414db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:52.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.911+0000 7f978a13e640 1 -- 172.21.15.186:0/734405568 shutdown_connections 2024-03-28T15:05:52.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:52.911+0000 7f978a13e640 1 -- 172.21.15.186:0/734405568 wait complete. 2024-03-28T15:05:52.923 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:05:54.924 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:05:55.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.036+0000 7fd6f5439640 1 Processor -- start 2024-03-28T15:05:55.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.036+0000 7fd6f5439640 1 -- start start 2024-03-28T15:05:55.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.037+0000 7fd6f5439640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f0154bd0 0x7fd6f015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:55.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.037+0000 7fd6f5439640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd6f00c53c0 0x7fd6f00c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:55.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.037+0000 7fd6f5439640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd6f00c57e0 0x7fd6f00cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:55.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.037+0000 7fd6f5439640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f00c5ca0 con 0x7fd6f00c57e0 2024-03-28T15:05:55.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.037+0000 7fd6f5439640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f015b080 con 0x7fd6f00c53c0 2024-03-28T15:05:55.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.037+0000 7fd6f5439640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f015b360 con 0x7fd6f0154bd0 2024-03-28T15:05:55.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.037+0000 7fd6ef7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd6f00c57e0 0x7fd6f00cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:55.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.037+0000 7fd6ef7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd6f00c57e0 0x7fd6f00cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52148/0 (socket says 172.21.15.186:52148) 2024-03-28T15:05:55.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.037+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/516199986 learned_addr learned my addr 172.21.15.186:0/516199986 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:05:55.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.037+0000 7fd6eeffd640 1 --2- 172.21.15.186:0/516199986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f0154bd0 0x7fd6f015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:55.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.037+0000 7fd6ee7fc640 1 --2- 172.21.15.186:0/516199986 >> v2:172.21.15.191:3300/0 conn(0x7fd6f00c53c0 0x7fd6f00c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:55.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.038+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/516199986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f0154bd0 msgr2=0x7fd6f015a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:05:55.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.038+0000 7fd6ef7fe640 1 --2- 172.21.15.186:0/516199986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f0154bd0 0x7fd6f015a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.038+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/516199986 >> v2:172.21.15.191:3300/0 conn(0x7fd6f00c53c0 msgr2=0x7fd6f00c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:05:55.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.038+0000 7fd6ef7fe640 1 --2- 172.21.15.186:0/516199986 >> v2:172.21.15.191:3300/0 conn(0x7fd6f00c53c0 0x7fd6f00c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.038+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/516199986 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f015b640 con 0x7fd6f00c57e0 2024-03-28T15:05:55.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.039+0000 7fd6ef7fe640 1 --2- 172.21.15.186:0/516199986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f00c57e0 0x7fd6f00cb040 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00cdbc0 tx=0x7fd6e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c2df325a8baff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:55.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.039+0000 7fd6edffb640 1 -- 172.21.15.186:0/516199986 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8034070 con 0x7fd6f00c57e0 2024-03-28T15:05:55.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.039+0000 7fd6edffb640 1 -- 172.21.15.186:0/516199986 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e8030860 con 0x7fd6f00c57e0 2024-03-28T15:05:55.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.039+0000 7fd6edffb640 1 -- 172.21.15.186:0/516199986 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8030c30 con 0x7fd6f00c57e0 2024-03-28T15:05:55.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.039+0000 7fd6f5439640 1 -- 172.21.15.186:0/516199986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f00c57e0 msgr2=0x7fd6f00cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:55.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.039+0000 7fd6f5439640 1 --2- 172.21.15.186:0/516199986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f00c57e0 0x7fd6f00cb040 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00cdbc0 tx=0x7fd6e802b9a0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.040+0000 7fd6f5439640 1 -- 172.21.15.186:0/516199986 shutdown_connections 2024-03-28T15:05:55.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.040+0000 7fd6f5439640 1 --2- 172.21.15.186:0/516199986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f00c57e0 0x7fd6f00cb040 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.040+0000 7fd6f5439640 1 --2- 172.21.15.186:0/516199986 >> v2:172.21.15.191:3300/0 conn(0x7fd6f00c53c0 0x7fd6f00c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.040+0000 7fd6f5439640 1 --2- 172.21.15.186:0/516199986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f0154bd0 0x7fd6f015a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.040+0000 7fd6f5439640 1 -- 172.21.15.186:0/516199986 >> 172.21.15.186:0/516199986 conn(0x7fd6f00731f0 msgr2=0x7fd6f0073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:55.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.040+0000 7fd6f5439640 1 -- 172.21.15.186:0/516199986 shutdown_connections 2024-03-28T15:05:55.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.040+0000 7fd6f5439640 1 -- 172.21.15.186:0/516199986 wait complete. 2024-03-28T15:05:55.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.041+0000 7fd6f5439640 1 Processor -- start 2024-03-28T15:05:55.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.041+0000 7fd6f5439640 1 -- start start 2024-03-28T15:05:55.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6f5439640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00c53c0 0x7fd6f0156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6f5439640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f00c57e0 0x7fd6f0156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6f5439640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154bd0 0x7fd6f014a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6f5439640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f00cc390 con 0x7fd6f0154bd0 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6f5439640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f00c61d0 con 0x7fd6f00c53c0 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6f5439640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f015ad10 con 0x7fd6f00c57e0 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6ef7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154bd0 0x7fd6f014a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6ef7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154bd0 0x7fd6f014a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52154/0 (socket says 172.21.15.186:52154) 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/4114004130 learned_addr learned my addr 172.21.15.186:0/4114004130 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6eeffd640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00c53c0 0x7fd6f0156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6ee7fc640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f00c57e0 0x7fd6f0156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/4114004130 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f00c57e0 msgr2=0x7fd6f0156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6ef7fe640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f00c57e0 0x7fd6f0156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/4114004130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00c53c0 msgr2=0x7fd6f0156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6ef7fe640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00c53c0 0x7fd6f0156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6ef7fe640 1 -- 172.21.15.186:0/4114004130 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f01e7b30 con 0x7fd6f0154bd0 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6eeffd640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00c53c0 0x7fd6f0156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6ef7fe640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154bd0 0x7fd6f014a9c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009960 tx=0x7fd6e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6d77fe640 1 -- 172.21.15.186:0/4114004130 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8048070 con 0x7fd6f0154bd0 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6d77fe640 1 -- 172.21.15.186:0/4114004130 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e8030680 con 0x7fd6f0154bd0 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6f5439640 1 -- 172.21.15.186:0/4114004130 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f00b4f70 con 0x7fd6f0154bd0 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.042+0000 7fd6d77fe640 1 -- 172.21.15.186:0/4114004130 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8034040 con 0x7fd6f0154bd0 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.043+0000 7fd6f5439640 1 -- 172.21.15.186:0/4114004130 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f00b5530 con 0x7fd6f0154bd0 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.043+0000 7fd6f5439640 1 -- 172.21.15.186:0/4114004130 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f0068ff0 con 0x7fd6f0154bd0 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.044+0000 7fd6d77fe640 1 -- 172.21.15.186:0/4114004130 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd6e804b070 con 0x7fd6f0154bd0 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.044+0000 7fd6d77fe640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 0x7fd6dc06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.044+0000 7fd6d77fe640 1 -- 172.21.15.186:0/4114004130 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7fd6e80ad830 con 0x7fd6f0154bd0 2024-03-28T15:05:55.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.044+0000 7fd6eeffd640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 0x7fd6dc06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:55.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.045+0000 7fd6eeffd640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 0x7fd6dc06ea20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009fd0 tx=0x7fd6e000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:55.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.049+0000 7fd6d77fe640 1 -- 172.21.15.186:0/4114004130 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd6f0068ff0 con 0x7fd6f0154bd0 2024-03-28T15:05:55.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.264+0000 7fd6f5439640 1 -- 172.21.15.186:0/4114004130 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6f0069500 con 0x7fd6f0154bd0 2024-03-28T15:05:55.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.265+0000 7fd6d77fe640 1 -- 172.21.15.186:0/4114004130 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6848 (secure 0 0 0) 0x7fd6e8030830 con 0x7fd6f0154bd0 2024-03-28T15:05:55.265 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:05:55.265 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:05:55.266 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 19 2024-03-28T15:05:55.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.266+0000 7fd6f5439640 1 -- 172.21.15.186:0/4114004130 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 msgr2=0x7fd6dc06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:55.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.266+0000 7fd6f5439640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 0x7fd6dc06ea20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009fd0 tx=0x7fd6e000b040 comp rx=0 tx=0).stop 2024-03-28T15:05:55.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.266+0000 7fd6f5439640 1 -- 172.21.15.186:0/4114004130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154bd0 msgr2=0x7fd6f014a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:55.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.266+0000 7fd6f5439640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154bd0 0x7fd6f014a9c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009960 tx=0x7fd6e8004730 comp rx=0 tx=0).stop 2024-03-28T15:05:55.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.267+0000 7fd6f5439640 1 -- 172.21.15.186:0/4114004130 shutdown_connections 2024-03-28T15:05:55.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.267+0000 7fd6f5439640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd6dc06c560 0x7fd6dc06ea20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.267+0000 7fd6f5439640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0154bd0 0x7fd6f014a9c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.267+0000 7fd6f5439640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6f00c57e0 0x7fd6f0156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.267+0000 7fd6f5439640 1 --2- 172.21.15.186:0/4114004130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6f00c53c0 0x7fd6f0156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:55.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.267+0000 7fd6f5439640 1 -- 172.21.15.186:0/4114004130 >> 172.21.15.186:0/4114004130 conn(0x7fd6f00731f0 msgr2=0x7fd6f00ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:55.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.267+0000 7fd6f5439640 1 -- 172.21.15.186:0/4114004130 shutdown_connections 2024-03-28T15:05:55.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:55.267+0000 7fd6f5439640 1 -- 172.21.15.186:0/4114004130 wait complete. 2024-03-28T15:05:55.278 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:05:56.383 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:05:56.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:05:57.279 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:05:57.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.388+0000 7f4513981640 1 Processor -- start 2024-03-28T15:05:57.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.388+0000 7f4513981640 1 -- start start 2024-03-28T15:05:57.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.389+0000 7f4513981640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f450c0733f0 0x7f450c0c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:57.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.389+0000 7f4513981640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450c0c1b40 con 0x7f450c1512b0 2024-03-28T15:05:57.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.389+0000 7f4513981640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450c0c15c0 con 0x7f450c150ed0 2024-03-28T15:05:57.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.389+0000 7f4513981640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f450c0c1e20 con 0x7f450c0733f0 2024-03-28T15:05:57.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.389+0000 7f4511f1e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f450c0733f0 0x7f450c0c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:57.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.389+0000 7f4510f1c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f450c1512b0 0x7f450c0b1750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:56588/0 (socket says 172.21.15.186:56588) 2024-03-28T15:05:57.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.389+0000 7f4510f1c640 1 -- 172.21.15.186:0/1103247142 learned_addr learned my addr 172.21.15.186:0/1103247142 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:05:57.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.389+0000 7f4511f1e640 1 -- 172.21.15.186:0/1103247142 >> v1:172.21.15.191:6789/0 conn(0x7f450c150ed0 legacy=0x7f450c156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:57.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.389+0000 7f4511f1e640 1 -- 172.21.15.186:0/1103247142 >> v1:172.21.15.186:6789/0 conn(0x7f450c1512b0 legacy=0x7f450c0b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:57.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.389+0000 7f4503fff640 1 -- 172.21.15.186:0/1103247142 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3844166967 0 0) 0x7f450c0c1b40 con 0x7f450c1512b0 2024-03-28T15:05:57.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.389+0000 7f4511f1e640 1 -- 172.21.15.186:0/1103247142 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450c072bf0 con 0x7f450c0733f0 2024-03-28T15:05:57.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.390+0000 7f4511f1e640 1 --2- 172.21.15.186:0/1103247142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f450c0733f0 0x7f450c0c0f40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f45040097a0 tx=0x7f450402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ad3d17b8576485 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:57.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.390+0000 7f4503fff640 1 -- 172.21.15.186:0/1103247142 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450402f810 con 0x7f450c0733f0 2024-03-28T15:05:57.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.390+0000 7f4503fff640 1 -- 172.21.15.186:0/1103247142 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f450402f9b0 con 0x7f450c0733f0 2024-03-28T15:05:57.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.390+0000 7f4503fff640 1 -- 172.21.15.186:0/1103247142 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450402fd80 con 0x7f450c0733f0 2024-03-28T15:05:57.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.391+0000 7f4513981640 1 -- 172.21.15.186:0/1103247142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f450c0733f0 msgr2=0x7f450c0c0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:57.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.391+0000 7f4513981640 1 --2- 172.21.15.186:0/1103247142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f450c0733f0 0x7f450c0c0f40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f45040097a0 tx=0x7f450402b9f0 comp rx=0 tx=0).stop 2024-03-28T15:05:57.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.391+0000 7f4513981640 1 -- 172.21.15.186:0/1103247142 shutdown_connections 2024-03-28T15:05:57.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.391+0000 7f4513981640 1 --2- 172.21.15.186:0/1103247142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f450c0733f0 0x7f450c0c0f40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:57.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.391+0000 7f4513981640 1 -- 172.21.15.186:0/1103247142 >> 172.21.15.186:0/1103247142 conn(0x7f450c0c8a20 msgr2=0x7f450c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:57.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.391+0000 7f4513981640 1 -- 172.21.15.186:0/1103247142 shutdown_connections 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.391+0000 7f4513981640 1 -- 172.21.15.186:0/1103247142 wait complete. 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.392+0000 7f4513981640 1 Processor -- start 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.392+0000 7f4513981640 1 -- start start 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4513981640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f450c0733f0 0x7f450c152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4513981640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f450c150ed0 0x7f450c1deeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4513981640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f450c1512b0 0x7f450c1e2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4513981640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f450c072f60 con 0x7f450c1512b0 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4513981640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f450c0c18a0 con 0x7f450c0733f0 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4513981640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f450c0b1e60 con 0x7f450c150ed0 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4511f1e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f450c1512b0 0x7f450c1e2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4511f1e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f450c1512b0 0x7f450c1e2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52164/0 (socket says 172.21.15.186:52164) 2024-03-28T15:05:57.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4511f1e640 1 -- 172.21.15.186:0/2939454972 learned_addr learned my addr 172.21.15.186:0/2939454972 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f451171d640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f450c0733f0 0x7f450c152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4510f1c640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f450c150ed0 0x7f450c1deeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4511f1e640 1 -- 172.21.15.186:0/2939454972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f450c150ed0 msgr2=0x7f450c1deeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4511f1e640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f450c150ed0 0x7f450c1deeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4511f1e640 1 -- 172.21.15.186:0/2939454972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f450c0733f0 msgr2=0x7f450c152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4511f1e640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f450c0733f0 0x7f450c152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4511f1e640 1 -- 172.21.15.186:0/2939454972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450c1f1660 con 0x7f450c1512b0 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f451171d640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f450c0733f0 0x7f450c152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.393+0000 7f4511f1e640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f450c1512b0 0x7f450c1e2970 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f450402c000 tx=0x7f4504004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.394+0000 7f4501ffb640 1 -- 172.21.15.186:0/2939454972 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4504046070 con 0x7f450c1512b0 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.394+0000 7f4501ffb640 1 -- 172.21.15.186:0/2939454972 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4504037bc0 con 0x7f450c1512b0 2024-03-28T15:05:57.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.394+0000 7f4501ffb640 1 -- 172.21.15.186:0/2939454972 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450402fe60 con 0x7f450c1512b0 2024-03-28T15:05:57.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.394+0000 7f4513981640 1 -- 172.21.15.186:0/2939454972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f450c1f1940 con 0x7f450c1512b0 2024-03-28T15:05:57.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.394+0000 7f4513981640 1 -- 172.21.15.186:0/2939454972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f450c1f1f00 con 0x7f450c1512b0 2024-03-28T15:05:57.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.395+0000 7f4513981640 1 -- 172.21.15.186:0/2939454972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f450c069500 con 0x7f450c1512b0 2024-03-28T15:05:57.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.395+0000 7f4501ffb640 1 -- 172.21.15.186:0/2939454972 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4504049070 con 0x7f450c1512b0 2024-03-28T15:05:57.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.396+0000 7f4501ffb640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f44f006c560 0x7f44f006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:57.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.396+0000 7f4501ffb640 1 -- 172.21.15.186:0/2939454972 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7f45040ad430 con 0x7f450c1512b0 2024-03-28T15:05:57.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.396+0000 7f451171d640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f44f006c560 0x7f44f006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:57.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.396+0000 7f451171d640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f44f006c560 0x7f44f006ea20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f45080012e0 tx=0x7f4508009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:57.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.400+0000 7f4501ffb640 1 -- 172.21.15.186:0/2939454972 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4504033030 con 0x7f450c1512b0 2024-03-28T15:05:57.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.612+0000 7f4513981640 1 -- 172.21.15.186:0/2939454972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f450c0c81e0 con 0x7f450c1512b0 2024-03-28T15:05:57.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.612+0000 7f4501ffb640 1 -- 172.21.15.186:0/2939454972 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6848 (secure 0 0 0) 0x7f450402f860 con 0x7f450c1512b0 2024-03-28T15:05:57.613 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:05:57.613 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:05:57.613 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 19 2024-03-28T15:05:57.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.614+0000 7f4513981640 1 -- 172.21.15.186:0/2939454972 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f44f006c560 msgr2=0x7f44f006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:57.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.614+0000 7f4513981640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f44f006c560 0x7f44f006ea20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f45080012e0 tx=0x7f4508009040 comp rx=0 tx=0).stop 2024-03-28T15:05:57.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.614+0000 7f4513981640 1 -- 172.21.15.186:0/2939454972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f450c1512b0 msgr2=0x7f450c1e2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:57.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.614+0000 7f4513981640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f450c1512b0 0x7f450c1e2970 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f450402c000 tx=0x7f4504004730 comp rx=0 tx=0).stop 2024-03-28T15:05:57.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.614+0000 7f4513981640 1 -- 172.21.15.186:0/2939454972 shutdown_connections 2024-03-28T15:05:57.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.614+0000 7f4513981640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f44f006c560 0x7f44f006ea20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:57.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.614+0000 7f4513981640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f450c1512b0 0x7f450c1e2970 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:57.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.614+0000 7f4513981640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f450c150ed0 0x7f450c1deeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:57.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.614+0000 7f4513981640 1 --2- 172.21.15.186:0/2939454972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f450c0733f0 0x7f450c152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:57.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.614+0000 7f4513981640 1 -- 172.21.15.186:0/2939454972 >> 172.21.15.186:0/2939454972 conn(0x7f450c0c8a20 msgr2=0x7f450c06ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:57.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.615+0000 7f4513981640 1 -- 172.21.15.186:0/2939454972 shutdown_connections 2024-03-28T15:05:57.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:57.615+0000 7f4513981640 1 -- 172.21.15.186:0/2939454972 wait complete. 2024-03-28T15:05:57.625 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:05:59.627 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:05:59.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.739+0000 7fdf91d02640 1 Processor -- start 2024-03-28T15:05:59.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.739+0000 7fdf91d02640 1 -- start start 2024-03-28T15:05:59.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf91d02640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf8c151990 0x7fdf8c0c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:59.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf91d02640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdf8c0c2d00 0x7fdf8c0c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:59.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf91d02640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf8c06caa0 con 0x7fdf8c0b1e30 2024-03-28T15:05:59.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf91d02640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fdf8c0c1da0 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf91d02640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fdf8c06cd80 con 0x7fdf8c151990 2024-03-28T15:05:59.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf8affd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fdf8c0b1e30 0x7fdf8c0712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:43980/0 (socket says 172.21.15.186:43980) 2024-03-28T15:05:59.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf8affd640 1 -- 172.21.15.186:0/3286405277 learned_addr learned my addr 172.21.15.186:0/3286405277 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:05:59.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf8bfff640 1 --2- 172.21.15.186:0/3286405277 >> v2:172.21.15.191:3300/0 conn(0x7fdf8c0c2d00 0x7fdf8c0c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:59.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf8b7fe640 1 --2- 172.21.15.186:0/3286405277 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf8c151990 0x7fdf8c0c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:59.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf8a7fc640 1 -- 172.21.15.186:0/3286405277 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1123100261 0 0) 0x7fdf8c06caa0 con 0x7fdf8c0b1e30 2024-03-28T15:05:59.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf8a7fc640 1 -- 172.21.15.186:0/3286405277 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf74003520 con 0x7fdf8c0b1e30 2024-03-28T15:05:59.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf8bfff640 1 -- 172.21.15.186:0/3286405277 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf8c151990 msgr2=0x7fdf8c0c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:05:59.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf8bfff640 1 --2- 172.21.15.186:0/3286405277 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf8c151990 0x7fdf8c0c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:59.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf8bfff640 1 -- 172.21.15.186:0/3286405277 >> v1:172.21.15.186:6789/0 conn(0x7fdf8c0b1e30 legacy=0x7fdf8c0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:59.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.740+0000 7fdf8bfff640 1 -- 172.21.15.186:0/3286405277 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8c0c1340 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.741+0000 7fdf8bfff640 1 --2- 172.21.15.186:0/3286405277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf8c0c2d00 0x7fdf8c0c2770 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdf84009d90 tx=0x7fdf8402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72335c08caf2f2fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:59.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.741+0000 7fdf8a7fc640 1 -- 172.21.15.186:0/3286405277 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84039070 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.741+0000 7fdf8a7fc640 1 -- 172.21.15.186:0/3286405277 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf8402cce0 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.741+0000 7fdf8a7fc640 1 -- 172.21.15.186:0/3286405277 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84034640 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.742+0000 7fdf91d02640 1 -- 172.21.15.186:0/3286405277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf8c0c2d00 msgr2=0x7fdf8c0c2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.742+0000 7fdf91d02640 1 --2- 172.21.15.186:0/3286405277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf8c0c2d00 0x7fdf8c0c2770 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdf84009d90 tx=0x7fdf8402bf30 comp rx=0 tx=0).stop 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.742+0000 7fdf91d02640 1 -- 172.21.15.186:0/3286405277 shutdown_connections 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.742+0000 7fdf91d02640 1 --2- 172.21.15.186:0/3286405277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf8c0c2d00 0x7fdf8c0c2770 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.742+0000 7fdf91d02640 1 --2- 172.21.15.186:0/3286405277 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf8c151990 0x7fdf8c0c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.742+0000 7fdf91d02640 1 -- 172.21.15.186:0/3286405277 >> 172.21.15.186:0/3286405277 conn(0x7fdf8c0c8a20 msgr2=0x7fdf8c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.742+0000 7fdf91d02640 1 -- 172.21.15.186:0/3286405277 shutdown_connections 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.742+0000 7fdf91d02640 1 -- 172.21.15.186:0/3286405277 wait complete. 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.743+0000 7fdf91d02640 1 Processor -- start 2024-03-28T15:05:59.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.743+0000 7fdf91d02640 1 -- start start 2024-03-28T15:05:59.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf91d02640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf8c0b1e30 0x7fdf8c1f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:59.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf91d02640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf8c0c2d00 0x7fdf8c0c52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:59.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf91d02640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf8c151990 0x7fdf8c1e3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:59.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf91d02640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8c073dd0 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf91d02640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8c0c1900 con 0x7fdf8c151990 2024-03-28T15:05:59.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf91d02640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fdf8c073f50 con 0x7fdf8c0b1e30 2024-03-28T15:05:59.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf8affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf8c0c2d00 0x7fdf8c0c52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:59.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf8affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf8c0c2d00 0x7fdf8c0c52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34376/0 (socket says 172.21.15.186:34376) 2024-03-28T15:05:59.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf8affd640 1 -- 172.21.15.186:0/1840313158 learned_addr learned my addr 172.21.15.186:0/1840313158 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:05:59.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf8bfff640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf8c151990 0x7fdf8c1e3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:59.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf8b7fe640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf8c0b1e30 0x7fdf8c1f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:59.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf8affd640 1 -- 172.21.15.186:0/1840313158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf8c0b1e30 msgr2=0x7fdf8c1f1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf8affd640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf8c0b1e30 0x7fdf8c1f1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf8affd640 1 -- 172.21.15.186:0/1840313158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf8c151990 msgr2=0x7fdf8c1e3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf8affd640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf8c151990 0x7fdf8c1e3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.744+0000 7fdf8affd640 1 -- 172.21.15.186:0/1840313158 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8c0c6240 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.745+0000 7fdf8affd640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf8c0c2d00 0x7fdf8c0c52f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdf80001eb0 tx=0x7fdf8000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.745+0000 7fdf73fff640 1 -- 172.21.15.186:0/1840313158 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80017070 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.745+0000 7fdf73fff640 1 -- 172.21.15.186:0/1840313158 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf80005d00 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.745+0000 7fdf73fff640 1 -- 172.21.15.186:0/1840313158 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80007310 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.745+0000 7fdf91d02640 1 -- 172.21.15.186:0/1840313158 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf8c0c3530 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.745+0000 7fdf91d02640 1 -- 172.21.15.186:0/1840313158 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf8c0cb540 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.746+0000 7fdf91d02640 1 -- 172.21.15.186:0/1840313158 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf8c069500 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.746+0000 7fdf73fff640 1 -- 172.21.15.186:0/1840313158 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdf8001a070 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.747+0000 7fdf73fff640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf7806c530 0x7fdf7806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:05:59.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.747+0000 7fdf73fff640 1 -- 172.21.15.186:0/1840313158 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7fdf8008abe0 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.747+0000 7fdf8b7fe640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf7806c530 0x7fdf7806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:05:59.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.747+0000 7fdf8b7fe640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf7806c530 0x7fdf7806e9f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009970 tx=0x7fdf7c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:05:59.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.751+0000 7fdf73fff640 1 -- 172.21.15.186:0/1840313158 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fdf80059130 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.963+0000 7fdf91d02640 1 -- 172.21.15.186:0/1840313158 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf8c14b950 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.963+0000 7fdf73fff640 1 -- 172.21.15.186:0/1840313158 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6848 (secure 0 0 0) 0x7fdf80058a30 con 0x7fdf8c0c2d00 2024-03-28T15:05:59.964 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:05:59.964 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:05:59.964 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 19 2024-03-28T15:05:59.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 -- 172.21.15.186:0/1840313158 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf7806c530 msgr2=0x7fdf7806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:59.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf7806c530 0x7fdf7806e9f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009970 tx=0x7fdf7c008040 comp rx=0 tx=0).stop 2024-03-28T15:05:59.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 -- 172.21.15.186:0/1840313158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf8c0c2d00 msgr2=0x7fdf8c0c52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:05:59.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf8c0c2d00 0x7fdf8c0c52f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdf80001eb0 tx=0x7fdf8000c820 comp rx=0 tx=0).stop 2024-03-28T15:05:59.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 -- 172.21.15.186:0/1840313158 shutdown_connections 2024-03-28T15:05:59.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf7806c530 0x7fdf7806e9f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:59.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf8c151990 0x7fdf8c1e3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:59.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf8c0c2d00 0x7fdf8c0c52f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:59.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 --2- 172.21.15.186:0/1840313158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf8c0b1e30 0x7fdf8c1f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:05:59.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 -- 172.21.15.186:0/1840313158 >> 172.21.15.186:0/1840313158 conn(0x7fdf8c0c8a20 msgr2=0x7fdf8c071700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:05:59.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 -- 172.21.15.186:0/1840313158 shutdown_connections 2024-03-28T15:05:59.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:05:59.966+0000 7fdf91d02640 1 -- 172.21.15.186:0/1840313158 wait complete. 2024-03-28T15:05:59.977 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:06:01.978 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:02.092 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.091+0000 7ff9fe519640 1 Processor -- start 2024-03-28T15:06:02.092 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.091+0000 7ff9fe519640 1 -- start start 2024-03-28T15:06:02.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.092+0000 7ff9fe519640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff9f8154bd0 0x7ff9f815a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:02.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.092+0000 7ff9fe519640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9f80c57e0 0x7ff9f80cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:02.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.092+0000 7ff9fe519640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9f80c6e00 con 0x7ff9f80c53c0 2024-03-28T15:06:02.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.092+0000 7ff9fe519640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff9f80c5ca0 con 0x7ff9f8154bd0 2024-03-28T15:06:02.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.092+0000 7ff9fe519640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f815b080 con 0x7ff9f80c57e0 2024-03-28T15:06:02.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.092+0000 7ff9f7fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff9f8154bd0 0x7ff9f815a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:02.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.092+0000 7ff9f77fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff9f80c53c0 0x7ff9f80c66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:43982/0 (socket says 172.21.15.186:43982) 2024-03-28T15:06:02.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.092+0000 7ff9f77fe640 1 -- 172.21.15.186:0/1780456097 learned_addr learned my addr 172.21.15.186:0/1780456097 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:06:02.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.093+0000 7ff9fcab6640 1 --2- 172.21.15.186:0/1780456097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9f80c57e0 0x7ff9f80cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:02.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.093+0000 7ff9f7fff640 1 -- 172.21.15.186:0/1780456097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9f80c57e0 msgr2=0x7ff9f80cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:02.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.093+0000 7ff9f7fff640 1 --2- 172.21.15.186:0/1780456097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9f80c57e0 0x7ff9f80cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:02.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.093+0000 7ff9f7fff640 1 -- 172.21.15.186:0/1780456097 >> v1:172.21.15.186:6789/0 conn(0x7ff9f80c53c0 legacy=0x7ff9f80c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:02.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.093+0000 7ff9f7fff640 1 -- 172.21.15.186:0/1780456097 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f815b360 con 0x7ff9f8154bd0 2024-03-28T15:06:02.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.093+0000 7ff9f6ffd640 1 -- 172.21.15.186:0/1780456097 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200976529 0 0) 0x7ff9f80c6e00 con 0x7ff9f80c53c0 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.093+0000 7ff9f7fff640 1 --2- 172.21.15.186:0/1780456097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8154bd0 0x7ff9f815a7d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0097a0 tx=0x7ff9ec02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42df3a9f8fb22bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.094+0000 7ff9f6ffd640 1 -- 172.21.15.186:0/1780456097 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec034070 con 0x7ff9f8154bd0 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.094+0000 7ff9f6ffd640 1 -- 172.21.15.186:0/1780456097 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ec030610 con 0x7ff9f8154bd0 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.094+0000 7ff9f6ffd640 1 -- 172.21.15.186:0/1780456097 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0309e0 con 0x7ff9f8154bd0 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.094+0000 7ff9fe519640 1 -- 172.21.15.186:0/1780456097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8154bd0 msgr2=0x7ff9f815a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.094+0000 7ff9fe519640 1 --2- 172.21.15.186:0/1780456097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8154bd0 0x7ff9f815a7d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0097a0 tx=0x7ff9ec02b960 comp rx=0 tx=0).stop 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.094+0000 7ff9fe519640 1 -- 172.21.15.186:0/1780456097 shutdown_connections 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.094+0000 7ff9fe519640 1 --2- 172.21.15.186:0/1780456097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9f80c57e0 0x7ff9f80cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.094+0000 7ff9fe519640 1 --2- 172.21.15.186:0/1780456097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8154bd0 0x7ff9f815a7d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.094+0000 7ff9fe519640 1 -- 172.21.15.186:0/1780456097 >> 172.21.15.186:0/1780456097 conn(0x7ff9f80731f0 msgr2=0x7ff9f8073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.094+0000 7ff9fe519640 1 -- 172.21.15.186:0/1780456097 shutdown_connections 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.094+0000 7ff9fe519640 1 -- 172.21.15.186:0/1780456097 wait complete. 2024-03-28T15:06:02.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.095+0000 7ff9fe519640 1 Processor -- start 2024-03-28T15:06:02.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.096+0000 7ff9fe519640 1 -- start start 2024-03-28T15:06:02.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.096+0000 7ff9fe519640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9f80c53c0 0x7ff9f80d0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:02.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.096+0000 7ff9fe519640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9f80c57e0 0x7ff9f81f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:02.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.096+0000 7ff9fe519640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8154bd0 0x7ff9f814a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:02.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.096+0000 7ff9fe519640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f80c61d0 con 0x7ff9f8154bd0 2024-03-28T15:06:02.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.096+0000 7ff9fe519640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f80c5ef0 con 0x7ff9f80c53c0 2024-03-28T15:06:02.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.096+0000 7ff9fe519640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f80cc390 con 0x7ff9f80c57e0 2024-03-28T15:06:02.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.096+0000 7ff9fcab6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8154bd0 0x7ff9f814a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:02.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.096+0000 7ff9fcab6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8154bd0 0x7ff9f814a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34400/0 (socket says 172.21.15.186:34400) 2024-03-28T15:06:02.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.096+0000 7ff9fcab6640 1 -- 172.21.15.186:0/2731392144 learned_addr learned my addr 172.21.15.186:0/2731392144 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:02.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.096+0000 7ff9f77fe640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9f80c57e0 0x7ff9f81f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:02.099 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9f7fff640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9f80c53c0 0x7ff9f80d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9fcab6640 1 -- 172.21.15.186:0/2731392144 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9f80c57e0 msgr2=0x7ff9f81f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9fcab6640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9f80c57e0 0x7ff9f81f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9fcab6640 1 -- 172.21.15.186:0/2731392144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9f80c53c0 msgr2=0x7ff9f80d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9fcab6640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9f80c53c0 0x7ff9f80d0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9fcab6640 1 -- 172.21.15.186:0/2731392144 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f80ca380 con 0x7ff9f8154bd0 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9fcab6640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8154bd0 0x7ff9f814a970 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0007e70 tx=0x7ff9f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9f4ff9640 1 -- 172.21.15.186:0/2731392144 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0014620 con 0x7ff9f8154bd0 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9f4ff9640 1 -- 172.21.15.186:0/2731392144 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f0019070 con 0x7ff9f8154bd0 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9fe519640 1 -- 172.21.15.186:0/2731392144 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9f80c94c0 con 0x7ff9f8154bd0 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9f4ff9640 1 -- 172.21.15.186:0/2731392144 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0014a20 con 0x7ff9f8154bd0 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.097+0000 7ff9fe519640 1 -- 172.21.15.186:0/2731392144 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9f815d850 con 0x7ff9f8154bd0 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.098+0000 7ff9fe519640 1 -- 172.21.15.186:0/2731392144 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9f8069500 con 0x7ff9f8154bd0 2024-03-28T15:06:02.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.099+0000 7ff9f4ff9640 1 -- 172.21.15.186:0/2731392144 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff9f0014bc0 con 0x7ff9f8154bd0 2024-03-28T15:06:02.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.099+0000 7ff9f4ff9640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff9e406c530 0x7ff9e406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:02.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.099+0000 7ff9f4ff9640 1 -- 172.21.15.186:0/2731392144 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7ff9f008c0b0 con 0x7ff9f8154bd0 2024-03-28T15:06:02.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.099+0000 7ff9f7fff640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff9e406c530 0x7ff9e406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:02.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.099+0000 7ff9f7fff640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff9e406c530 0x7ff9e406e9f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec036000 tx=0x7ff9ec03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:02.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.103+0000 7ff9f4ff9640 1 -- 172.21.15.186:0/2731392144 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff9f0059eb0 con 0x7ff9f8154bd0 2024-03-28T15:06:02.317 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.316+0000 7ff9fe519640 1 -- 172.21.15.186:0/2731392144 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9f80cbb60 con 0x7ff9f8154bd0 2024-03-28T15:06:02.317 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.317+0000 7ff9f4ff9640 1 -- 172.21.15.186:0/2731392144 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6848 (secure 0 0 0) 0x7ff9f005e1b0 con 0x7ff9f8154bd0 2024-03-28T15:06:02.318 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:02.318 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:06:02.318 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 19 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 -- 172.21.15.186:0/2731392144 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff9e406c530 msgr2=0x7ff9e406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff9e406c530 0x7ff9e406e9f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec036000 tx=0x7ff9ec03b040 comp rx=0 tx=0).stop 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 -- 172.21.15.186:0/2731392144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8154bd0 msgr2=0x7ff9f814a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8154bd0 0x7ff9f814a970 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0007e70 tx=0x7ff9f000e4b0 comp rx=0 tx=0).stop 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 -- 172.21.15.186:0/2731392144 shutdown_connections 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff9e406c530 0x7ff9e406e9f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9f8154bd0 0x7ff9f814a970 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9f80c57e0 0x7ff9f81f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 --2- 172.21.15.186:0/2731392144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9f80c53c0 0x7ff9f80d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 -- 172.21.15.186:0/2731392144 >> 172.21.15.186:0/2731392144 conn(0x7ff9f80731f0 msgr2=0x7ff9f815a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 -- 172.21.15.186:0/2731392144 shutdown_connections 2024-03-28T15:06:02.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:02.319+0000 7ff9fe519640 1 -- 172.21.15.186:0/2731392144 wait complete. 2024-03-28T15:06:02.331 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:06:04.333 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:04.443 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.442+0000 7f5851743640 1 Processor -- start 2024-03-28T15:06:04.443 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.442+0000 7f5851743640 1 -- start start 2024-03-28T15:06:04.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.443+0000 7f5851743640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c0c2d00 0x7f584c0719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:04.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.443+0000 7f5851743640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f584c06caa0 con 0x7f584c0b1e30 2024-03-28T15:06:04.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.443+0000 7f5851743640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f584c06cd80 con 0x7f584c151990 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.443+0000 7f5851743640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f584c0c1da0 con 0x7f584c0c2d00 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.443+0000 7f584a7fc640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f584c0b1e30 0x7f584c0712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:43994/0 (socket says 172.21.15.186:43994) 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.443+0000 7f584a7fc640 1 -- 172.21.15.186:0/3754172663 learned_addr learned my addr 172.21.15.186:0/3754172663 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f584b7fe640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c0c2d00 0x7f584c0719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2982536329 0 0) 0x7f584c06caa0 con 0x7f584c0b1e30 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5834003520 con 0x7f584c0b1e30 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4128927155 0 0) 0x7f584c06cd80 con 0x7f584c151990 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f584c06caa0 con 0x7f584c151990 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3228017354 0 0) 0x7f5834003520 con 0x7f584c0b1e30 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c0c2d00 msgr2=0x7f584c0719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c0c2d00 0x7f584c0719c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 >> v1:172.21.15.191:6789/0 conn(0x7f584c151990 legacy=0x7f584c0c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c0c1340 con 0x7f584c0b1e30 2024-03-28T15:06:04.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f5840001670 con 0x7f584c0b1e30 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f584b7fe640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c0c2d00 0x7f584c0719c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 >> v1:172.21.15.186:6789/0 conn(0x7f584c0b1e30 legacy=0x7f584c0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5834003b70 0x7f5834003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5834004800 0x7f5834004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c0c2d00 0x7f58340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f584c0c1da0 con 0x7f5834004800 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f583400b9f0 con 0x7f5834003b70 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f583400bbf0 con 0x7f584c0c2d00 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f584a7fc640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5834004800 0x7f5834004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f584b7fe640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c0c2d00 0x7f58340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f584affd640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5834003b70 0x7f5834003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f584a7fc640 1 -- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c0c2d00 msgr2=0x7f58340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f584a7fc640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c0c2d00 0x7f58340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:04.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f584a7fc640 1 -- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5834003b70 msgr2=0x7f5834003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f584a7fc640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5834003b70 0x7f5834003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.444+0000 7f584a7fc640 1 -- 172.21.15.186:0/3754172663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5834003520 con 0x7f5834004800 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.445+0000 7f584a7fc640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5834004800 0x7f5834004c80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f584c0b2250 tx=0x7f5834005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b4407809183470a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.445+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5840007ce0 con 0x7f5834004800 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.445+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5840030070 con 0x7f5834004800 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.445+0000 7f5849ffb640 1 -- 172.21.15.186:0/3754172663 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5840005d50 con 0x7f5834004800 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.446+0000 7f5851743640 1 -- 172.21.15.186:0/3754172663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5834004800 msgr2=0x7f5834004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.446+0000 7f5851743640 1 --2- 172.21.15.186:0/3754172663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5834004800 0x7f5834004c80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f584c0b2250 tx=0x7f5834005830 comp rx=0 tx=0).stop 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.446+0000 7f584b7fe640 1 -- 172.21.15.186:0/3754172663 reap_dead start 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.446+0000 7f5851743640 1 -- 172.21.15.186:0/3754172663 shutdown_connections 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.446+0000 7f5851743640 1 -- 172.21.15.186:0/3754172663 >> 172.21.15.186:0/3754172663 conn(0x7f584c0c8a20 msgr2=0x7f584c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.446+0000 7f5851743640 1 -- 172.21.15.186:0/3754172663 shutdown_connections 2024-03-28T15:06:04.449 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.446+0000 7f5851743640 1 -- 172.21.15.186:0/3754172663 wait complete. 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.447+0000 7f5851743640 1 Processor -- start 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.447+0000 7f5851743640 1 -- start start 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f5851743640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5834004800 0x7f584c0ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f5851743640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f584c0cb960 0x7f584c144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f5851743640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c1f13e0 0x7f584c1470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f5851743640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f584c0c2940 con 0x7f584c0cb960 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f5851743640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f584c0c1900 con 0x7f5834004800 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f5851743640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f584c0c2ac0 con 0x7f584c1f13e0 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f584a7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f584c0cb960 0x7f584c144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f584a7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f584c0cb960 0x7f584c144960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34422/0 (socket says 172.21.15.186:34422) 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f584a7fc640 1 -- 172.21.15.186:0/1196273631 learned_addr learned my addr 172.21.15.186:0/1196273631 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:04.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f584affd640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5834004800 0x7f584c0ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f584b7fe640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c1f13e0 0x7f584c1470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f584a7fc640 1 -- 172.21.15.186:0/1196273631 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c1f13e0 msgr2=0x7f584c1470f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f584a7fc640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c1f13e0 0x7f584c1470f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f584a7fc640 1 -- 172.21.15.186:0/1196273631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5834004800 msgr2=0x7f584c0ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f584a7fc640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5834004800 0x7f584c0ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.448+0000 7f584a7fc640 1 -- 172.21.15.186:0/1196273631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c144200 con 0x7f584c0cb960 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.449+0000 7f584a7fc640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f584c0cb960 0x7f584c144960 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5840007fd0 tx=0x7f5840001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.449+0000 7f58337fe640 1 -- 172.21.15.186:0/1196273631 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5840007ce0 con 0x7f584c0cb960 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.449+0000 7f58337fe640 1 -- 172.21.15.186:0/1196273631 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f584002cbf0 con 0x7f584c0cb960 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.449+0000 7f5851743640 1 -- 172.21.15.186:0/1196273631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584c144750 con 0x7f584c0cb960 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.449+0000 7f58337fe640 1 -- 172.21.15.186:0/1196273631 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5840001c10 con 0x7f584c0cb960 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.449+0000 7f5851743640 1 -- 172.21.15.186:0/1196273631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f584c14dab0 con 0x7f584c0cb960 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.450+0000 7f5851743640 1 -- 172.21.15.186:0/1196273631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f584c069500 con 0x7f584c0cb960 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.450+0000 7f58337fe640 1 -- 172.21.15.186:0/1196273631 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5840007070 con 0x7f584c0cb960 2024-03-28T15:06:04.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.451+0000 7f58337fe640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f583806c500 0x7f583806e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:04.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.451+0000 7f58337fe640 1 -- 172.21.15.186:0/1196273631 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7f58400acab0 con 0x7f584c0cb960 2024-03-28T15:06:04.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.451+0000 7f584affd640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f583806c500 0x7f583806e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:04.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.451+0000 7f584affd640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f583806c500 0x7f583806e9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f583c001fd0 tx=0x7f583c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:04.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.455+0000 7f58337fe640 1 -- 172.21.15.186:0/1196273631 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f584007a9d0 con 0x7f584c0cb960 2024-03-28T15:06:04.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.668+0000 7f5851743640 1 -- 172.21.15.186:0/1196273631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f584c069500 con 0x7f584c0cb960 2024-03-28T15:06:04.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.669+0000 7f58337fe640 1 -- 172.21.15.186:0/1196273631 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6848 (secure 0 0 0) 0x7f584007ecd0 con 0x7f584c0cb960 2024-03-28T15:06:04.669 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:04.669 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:04:10.474185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"f","rank":0,"incarnation":14,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6837/3828930138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3828930138},{"type":"v1","addr":"172.21.15.191:6837","nonce":3828930138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1}]} 2024-03-28T15:06:04.669 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 19 2024-03-28T15:06:04.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 -- 172.21.15.186:0/1196273631 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f583806c500 msgr2=0x7f583806e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:04.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f583806c500 0x7f583806e9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f583c001fd0 tx=0x7f583c009040 comp rx=0 tx=0).stop 2024-03-28T15:06:04.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 -- 172.21.15.186:0/1196273631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f584c0cb960 msgr2=0x7f584c144960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:04.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f584c0cb960 0x7f584c144960 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5840007fd0 tx=0x7f5840001a30 comp rx=0 tx=0).stop 2024-03-28T15:06:04.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 -- 172.21.15.186:0/1196273631 shutdown_connections 2024-03-28T15:06:04.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f583806c500 0x7f583806e9c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:04.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f584c1f13e0 0x7f584c1470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:04.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f584c0cb960 0x7f584c144960 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:04.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 --2- 172.21.15.186:0/1196273631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5834004800 0x7f584c0ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:04.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 -- 172.21.15.186:0/1196273631 >> 172.21.15.186:0/1196273631 conn(0x7f584c0c8a20 msgr2=0x7f584c0c7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:04.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 -- 172.21.15.186:0/1196273631 shutdown_connections 2024-03-28T15:06:04.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:04.671+0000 7f5851743640 1 -- 172.21.15.186:0/1196273631 wait complete. 2024-03-28T15:06:04.682 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:06:06.683 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:06.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.796+0000 7fd8154a4640 1 Processor -- start 2024-03-28T15:06:06.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.796+0000 7fd8154a4640 1 -- start start 2024-03-28T15:06:06.797 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.796+0000 7fd8154a4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd810154bd0 0x7fd81015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:06.797 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.796+0000 7fd8154a4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd8100c53c0 0x7fd8100c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.796+0000 7fd8154a4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd81015b080 con 0x7fd8100c57e0 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.796+0000 7fd8154a4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd8100c5ca0 con 0x7fd8100c53c0 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.796+0000 7fd8154a4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd81015b360 con 0x7fd810154bd0 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.797+0000 7fd80e7fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd8100c53c0 0x7fd8100c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.797+0000 7fd80e7fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd8100c53c0 0x7fd8100c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34432/0 (socket says 172.21.15.186:34432) 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.797+0000 7fd80e7fc640 1 -- 172.21.15.186:0/1401824694 learned_addr learned my addr 172.21.15.186:0/1401824694 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.797+0000 7fd80effd640 1 --2- 172.21.15.186:0/1401824694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd810154bd0 0x7fd81015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.797+0000 7fd80e7fc640 1 -- 172.21.15.186:0/1401824694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd810154bd0 msgr2=0x7fd81015a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.797+0000 7fd80e7fc640 1 --2- 172.21.15.186:0/1401824694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd810154bd0 0x7fd81015a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.797+0000 7fd80e7fc640 1 -- 172.21.15.186:0/1401824694 >> v1:172.21.15.191:6789/0 conn(0x7fd8100c57e0 legacy=0x7fd8100cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.797+0000 7fd80e7fc640 1 -- 172.21.15.186:0/1401824694 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81015b640 con 0x7fd8100c53c0 2024-03-28T15:06:06.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.797+0000 7fd80effd640 1 --2- 172.21.15.186:0/1401824694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd810154bd0 0x7fd81015a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.798+0000 7fd80e7fc640 1 --2- 172.21.15.186:0/1401824694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8100c53c0 0x7fd8100c66f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd804009870 tx=0x7fd80402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6549a63d775d366b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.798+0000 7fd80dffb640 1 -- 172.21.15.186:0/1401824694 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80402f830 con 0x7fd8100c53c0 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.798+0000 7fd80dffb640 1 -- 172.21.15.186:0/1401824694 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80402f9d0 con 0x7fd8100c53c0 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.798+0000 7fd80dffb640 1 -- 172.21.15.186:0/1401824694 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80402fda0 con 0x7fd8100c53c0 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.798+0000 7fd8154a4640 1 -- 172.21.15.186:0/1401824694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8100c53c0 msgr2=0x7fd8100c66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.798+0000 7fd8154a4640 1 --2- 172.21.15.186:0/1401824694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8100c53c0 0x7fd8100c66f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd804009870 tx=0x7fd80402ba10 comp rx=0 tx=0).stop 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.798+0000 7fd8154a4640 1 -- 172.21.15.186:0/1401824694 shutdown_connections 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.798+0000 7fd8154a4640 1 --2- 172.21.15.186:0/1401824694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8100c53c0 0x7fd8100c66f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.798+0000 7fd8154a4640 1 --2- 172.21.15.186:0/1401824694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd810154bd0 0x7fd81015a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.798+0000 7fd8154a4640 1 -- 172.21.15.186:0/1401824694 >> 172.21.15.186:0/1401824694 conn(0x7fd8100731f0 msgr2=0x7fd810073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.798+0000 7fd8154a4640 1 -- 172.21.15.186:0/1401824694 shutdown_connections 2024-03-28T15:06:06.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.799+0000 7fd8154a4640 1 -- 172.21.15.186:0/1401824694 wait complete. 2024-03-28T15:06:06.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.800+0000 7fd8154a4640 1 Processor -- start 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.800+0000 7fd8154a4640 1 -- start start 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.800+0000 7fd8154a4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8100c53c0 0x7fd8100b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.800+0000 7fd8154a4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8100c57e0 0x7fd81015e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.800+0000 7fd8154a4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd810154bd0 0x7fd8101f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.800+0000 7fd8154a4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd8100cc390 con 0x7fd810154bd0 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.800+0000 7fd8154a4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd8100c61d0 con 0x7fd8100c57e0 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.800+0000 7fd8154a4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd81015ad10 con 0x7fd8100c53c0 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd80f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd810154bd0 0x7fd8101f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd80f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd810154bd0 0x7fd8101f3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34436/0 (socket says 172.21.15.186:34436) 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd80f7fe640 1 -- 172.21.15.186:0/556553296 learned_addr learned my addr 172.21.15.186:0/556553296 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd80e7fc640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8100c57e0 0x7fd81015e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd80effd640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8100c53c0 0x7fd8100b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd80f7fe640 1 -- 172.21.15.186:0/556553296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8100c53c0 msgr2=0x7fd8100b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:06.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd80f7fe640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8100c53c0 0x7fd8100b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:06.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd80f7fe640 1 -- 172.21.15.186:0/556553296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8100c57e0 msgr2=0x7fd81015e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:06.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd80f7fe640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8100c57e0 0x7fd81015e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:06.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd80f7fe640 1 -- 172.21.15.186:0/556553296 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8100b5620 con 0x7fd810154bd0 2024-03-28T15:06:06.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd80f7fe640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd810154bd0 0x7fd8101f3d50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd80800c7d0 tx=0x7fd80800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:06.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd7f77fe640 1 -- 172.21.15.186:0/556553296 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd808003b70 con 0x7fd810154bd0 2024-03-28T15:06:06.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd7f77fe640 1 -- 172.21.15.186:0/556553296 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd808003d30 con 0x7fd810154bd0 2024-03-28T15:06:06.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd8154a4640 1 -- 172.21.15.186:0/556553296 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8101f4c30 con 0x7fd810154bd0 2024-03-28T15:06:06.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd7f77fe640 1 -- 172.21.15.186:0/556553296 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80800aa90 con 0x7fd810154bd0 2024-03-28T15:06:06.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.801+0000 7fd8154a4640 1 -- 172.21.15.186:0/556553296 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8101f4950 con 0x7fd810154bd0 2024-03-28T15:06:06.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.803+0000 7fd8154a4640 1 -- 172.21.15.186:0/556553296 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8100ced20 con 0x7fd810154bd0 2024-03-28T15:06:06.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.803+0000 7fd7f77fe640 1 -- 172.21.15.186:0/556553296 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd80800ac30 con 0x7fd810154bd0 2024-03-28T15:06:06.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.803+0000 7fd7f77fe640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7fc06c530 0x7fd7fc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:06.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.803+0000 7fd7f77fe640 1 -- 172.21.15.186:0/556553296 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7fd80808b690 con 0x7fd810154bd0 2024-03-28T15:06:06.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.803+0000 7fd80effd640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7fc06c530 0x7fd7fc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:06.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.804+0000 7fd80effd640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7fc06c530 0x7fd7fc06e9f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd800006fd0 tx=0x7fd800008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:06.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:06.808+0000 7fd7f77fe640 1 -- 172.21.15.186:0/556553296 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd8100ced20 con 0x7fd810154bd0 2024-03-28T15:06:07.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.019+0000 7fd8154a4640 1 -- 172.21.15.186:0/556553296 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd810069500 con 0x7fd810154bd0 2024-03-28T15:06:07.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.019+0000 7fd7f77fe640 1 -- 172.21.15.186:0/556553296 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6012 (secure 0 0 0) 0x7fd8080594e0 con 0x7fd810154bd0 2024-03-28T15:06:07.020 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:07.020 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:06.202561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:06:07.020 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 20 2024-03-28T15:06:07.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 -- 172.21.15.186:0/556553296 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7fc06c530 msgr2=0x7fd7fc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:07.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7fc06c530 0x7fd7fc06e9f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd800006fd0 tx=0x7fd800008040 comp rx=0 tx=0).stop 2024-03-28T15:06:07.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 -- 172.21.15.186:0/556553296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd810154bd0 msgr2=0x7fd8101f3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:07.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd810154bd0 0x7fd8101f3d50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd80800c7d0 tx=0x7fd80800cc90 comp rx=0 tx=0).stop 2024-03-28T15:06:07.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 -- 172.21.15.186:0/556553296 shutdown_connections 2024-03-28T15:06:07.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd7fc06c530 0x7fd7fc06e9f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:07.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd810154bd0 0x7fd8101f3d50 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:07.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8100c57e0 0x7fd81015e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:07.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 --2- 172.21.15.186:0/556553296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8100c53c0 0x7fd8100b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:07.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 -- 172.21.15.186:0/556553296 >> 172.21.15.186:0/556553296 conn(0x7fd8100731f0 msgr2=0x7fd810158a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:07.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 -- 172.21.15.186:0/556553296 shutdown_connections 2024-03-28T15:06:07.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.021+0000 7fd8154a4640 1 -- 172.21.15.186:0/556553296 wait complete. 2024-03-28T15:06:07.033 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-03-28T15:06:07.033 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-28T15:06:07.033 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:07.133 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.133+0000 7f417daed640 1 Processor -- start 2024-03-28T15:06:07.133 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.133+0000 7f417daed640 1 -- start start 2024-03-28T15:06:07.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.133+0000 7f417daed640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41780b1e30 0x7f41780712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:07.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.133+0000 7f417daed640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f417806caa0 con 0x7f4178151990 2024-03-28T15:06:07.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.134+0000 7f417daed640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f417806cd80 con 0x7f41780c2d00 2024-03-28T15:06:07.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.134+0000 7f417daed640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f41780c1da0 con 0x7f41780b1e30 2024-03-28T15:06:07.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.134+0000 7f4176ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41780b1e30 0x7f41780712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:07.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.134+0000 7f41777fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4178151990 0x7f41780c2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:44006/0 (socket says 172.21.15.186:44006) 2024-03-28T15:06:07.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.134+0000 7f41777fe640 1 -- 172.21.15.186:0/1266733116 learned_addr learned my addr 172.21.15.186:0/1266733116 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:06:07.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.134+0000 7f4176ffd640 1 -- 172.21.15.186:0/1266733116 >> v1:172.21.15.191:6789/0 conn(0x7f41780c2d00 legacy=0x7f41780717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:07.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.134+0000 7f4176ffd640 1 -- 172.21.15.186:0/1266733116 >> v1:172.21.15.186:6789/0 conn(0x7f4178151990 legacy=0x7f41780c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:07.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.134+0000 7f4176ffd640 1 -- 172.21.15.186:0/1266733116 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41780c1340 con 0x7f41780b1e30 2024-03-28T15:06:07.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.134+0000 7f41767fc640 1 -- 172.21.15.186:0/1266733116 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1074231530 0 0) 0x7f417806caa0 con 0x7f4178151990 2024-03-28T15:06:07.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.135+0000 7f4176ffd640 1 --2- 172.21.15.186:0/1266733116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41780b1e30 0x7f41780712b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f416c0097a0 tx=0x7f416c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e4367ccfdc2ee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:07.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.135+0000 7f41767fc640 1 -- 172.21.15.186:0/1266733116 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c034070 con 0x7f41780b1e30 2024-03-28T15:06:07.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.135+0000 7f41767fc640 1 -- 172.21.15.186:0/1266733116 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f416c030610 con 0x7f41780b1e30 2024-03-28T15:06:07.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.135+0000 7f41767fc640 1 -- 172.21.15.186:0/1266733116 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c0309e0 con 0x7f41780b1e30 2024-03-28T15:06:07.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.135+0000 7f417daed640 1 -- 172.21.15.186:0/1266733116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41780b1e30 msgr2=0x7f41780712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:07.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.135+0000 7f417daed640 1 --2- 172.21.15.186:0/1266733116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41780b1e30 0x7f41780712b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f416c0097a0 tx=0x7f416c02b960 comp rx=0 tx=0).stop 2024-03-28T15:06:07.137 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.135+0000 7f417daed640 1 -- 172.21.15.186:0/1266733116 shutdown_connections 2024-03-28T15:06:07.137 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.135+0000 7f417daed640 1 --2- 172.21.15.186:0/1266733116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41780b1e30 0x7f41780712b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:07.137 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.135+0000 7f417daed640 1 -- 172.21.15.186:0/1266733116 >> 172.21.15.186:0/1266733116 conn(0x7f41780c8a20 msgr2=0x7f41780c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:07.137 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.135+0000 7f417daed640 1 -- 172.21.15.186:0/1266733116 shutdown_connections 2024-03-28T15:06:07.137 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.135+0000 7f417daed640 1 -- 172.21.15.186:0/1266733116 wait complete. 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.136+0000 7f417daed640 1 Processor -- start 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.137+0000 7f417daed640 1 -- start start 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.137+0000 7f417daed640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41780b1e30 0x7f4178152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.137+0000 7f417daed640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41780c2d00 0x7f4178134550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.137+0000 7f417daed640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4178151990 0x7f4178134a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.137+0000 7f417daed640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41780c2940 con 0x7f41780b1e30 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.137+0000 7f417daed640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f41780c1900 con 0x7f4178151990 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.137+0000 7f417daed640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f41780c2ac0 con 0x7f41780c2d00 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.137+0000 7f41777fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41780b1e30 0x7f4178152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.137+0000 7f41777fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41780b1e30 0x7f4178152970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34446/0 (socket says 172.21.15.186:34446) 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.137+0000 7f41777fe640 1 -- 172.21.15.186:0/1967564392 learned_addr learned my addr 172.21.15.186:0/1967564392 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f4176ffd640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41780c2d00 0x7f4178134550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:07.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f4177fff640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4178151990 0x7f4178134a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f41777fe640 1 -- 172.21.15.186:0/1967564392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41780c2d00 msgr2=0x7f4178134550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f41777fe640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41780c2d00 0x7f4178134550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f41777fe640 1 -- 172.21.15.186:0/1967564392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4178151990 msgr2=0x7f4178134a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f41777fe640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4178151990 0x7f4178134a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f41777fe640 1 -- 172.21.15.186:0/1967564392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417814c070 con 0x7f41780b1e30 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f41777fe640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41780b1e30 0x7f4178152970 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f416800c4d0 tx=0x7f416800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f415ffff640 1 -- 172.21.15.186:0/1967564392 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168018070 con 0x7f41780b1e30 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f415ffff640 1 -- 172.21.15.186:0/1967564392 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4168007500 con 0x7f41780b1e30 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f417daed640 1 -- 172.21.15.186:0/1967564392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f417814c350 con 0x7f41780b1e30 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f415ffff640 1 -- 172.21.15.186:0/1967564392 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168004a20 con 0x7f41780b1e30 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.138+0000 7f417daed640 1 -- 172.21.15.186:0/1967564392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f417814c910 con 0x7f41780b1e30 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.139+0000 7f417daed640 1 -- 172.21.15.186:0/1967564392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4178069500 con 0x7f41780b1e30 2024-03-28T15:06:07.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.140+0000 7f415ffff640 1 -- 172.21.15.186:0/1967564392 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f416801b070 con 0x7f41780b1e30 2024-03-28T15:06:07.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.140+0000 7f415ffff640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f416406c530 0x7f416406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:07.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.140+0000 7f4176ffd640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f416406c530 0x7f416406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:07.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.140+0000 7f415ffff640 1 -- 172.21.15.186:0/1967564392 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f416808ac50 con 0x7f41780b1e30 2024-03-28T15:06:07.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.140+0000 7f4176ffd640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f416406c530 0x7f416406e9f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f416c036000 tx=0x7f416c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:07.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.144+0000 7f415ffff640 1 -- 172.21.15.186:0/1967564392 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4168090020 con 0x7f41780b1e30 2024-03-28T15:06:07.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.360+0000 7f417daed640 1 -- 172.21.15.186:0/1967564392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41780c1080 con 0x7f41780b1e30 2024-03-28T15:06:07.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.361+0000 7f415ffff640 1 -- 172.21.15.186:0/1967564392 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6012 (secure 0 0 0) 0x7f4168059060 con 0x7f41780b1e30 2024-03-28T15:06:07.361 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:07.361 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:06.202561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:06:07.361 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 20 2024-03-28T15:06:07.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 -- 172.21.15.186:0/1967564392 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f416406c530 msgr2=0x7f416406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:07.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f416406c530 0x7f416406e9f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f416c036000 tx=0x7f416c008030 comp rx=0 tx=0).stop 2024-03-28T15:06:07.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 -- 172.21.15.186:0/1967564392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41780b1e30 msgr2=0x7f4178152970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:07.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41780b1e30 0x7f4178152970 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f416800c4d0 tx=0x7f416800c990 comp rx=0 tx=0).stop 2024-03-28T15:06:07.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 -- 172.21.15.186:0/1967564392 shutdown_connections 2024-03-28T15:06:07.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f416406c530 0x7f416406e9f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:07.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4178151990 0x7f4178134a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:07.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41780c2d00 0x7f4178134550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:07.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 --2- 172.21.15.186:0/1967564392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41780b1e30 0x7f4178152970 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:07.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 -- 172.21.15.186:0/1967564392 >> 172.21.15.186:0/1967564392 conn(0x7f41780c8a20 msgr2=0x7f417814fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:07.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 -- 172.21.15.186:0/1967564392 shutdown_connections 2024-03-28T15:06:07.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:07.363+0000 7f417daed640 1 -- 172.21.15.186:0/1967564392 wait complete. 2024-03-28T15:06:07.375 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 20, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T15:01:31.425299+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 20, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4790": { "addr": "172.21.15.191:6834/1992706523", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 1992706523, "type": "v2" }, { "addr": "172.21.15.191:6834", "nonce": 1992706523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4790, "incarnation": 20, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T15:06:06.202561+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4790 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/1278716016", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6833", "nonce": 1278716016, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1278716016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6835/2068761001", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2068761001, "type": "v2" }, { "addr": "172.21.15.186:6835", "nonce": 2068761001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2344737463", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6837", "nonce": 2344737463, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2344737463, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4284, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/1548903381", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 1548903381, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 1548903381, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T15:06:09.376 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:09.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.484+0000 7f4fc455e640 1 Processor -- start 2024-03-28T15:06:09.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.484+0000 7f4fc455e640 1 -- start start 2024-03-28T15:06:09.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.485+0000 7f4fc455e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4fbc0c53c0 0x7f4fbc0c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:09.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.485+0000 7f4fc455e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 0x7f4fbc0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:09.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.485+0000 7f4fc455e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fbc15b080 con 0x7f4fbc154bd0 2024-03-28T15:06:09.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.485+0000 7f4fc455e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f4fbc0c5ca0 con 0x7f4fbc0c53c0 2024-03-28T15:06:09.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.485+0000 7f4fc455e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4fbc15b360 con 0x7f4fbc0c57e0 2024-03-28T15:06:09.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.486+0000 7f4fc2afb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 0x7f4fbc0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:09.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.486+0000 7f4fc1af9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4fbc0c53c0 0x7f4fbc0c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:09.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.486+0000 7f4fc22fa640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4fbc154bd0 0x7f4fbc15a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:44282/0 (socket says 172.21.15.186:44282) 2024-03-28T15:06:09.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.486+0000 7f4fc2afb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 0x7f4fbc0cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:58918/0 (socket says 172.21.15.186:58918) 2024-03-28T15:06:09.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.486+0000 7f4fc22fa640 1 -- 172.21.15.186:0/2057333459 learned_addr learned my addr 172.21.15.186:0/2057333459 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:06:09.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.486+0000 7f4fc2afb640 1 -- 172.21.15.186:0/2057333459 >> v2:172.21.15.191:3300/0 conn(0x7f4fbc0c53c0 msgr2=0x7f4fbc0c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:09.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.486+0000 7f4fc2afb640 1 --2- 172.21.15.186:0/2057333459 >> v2:172.21.15.191:3300/0 conn(0x7f4fbc0c53c0 0x7f4fbc0c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:09.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.486+0000 7f4fc2afb640 1 -- 172.21.15.186:0/2057333459 >> v1:172.21.15.191:6789/0 conn(0x7f4fbc154bd0 legacy=0x7f4fbc15a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:09.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.486+0000 7f4fc2afb640 1 -- 172.21.15.186:0/2057333459 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fbc15b640 con 0x7f4fbc0c57e0 2024-03-28T15:06:09.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.486+0000 7f4fc1af9640 1 --2- 172.21.15.186:0/2057333459 >> v2:172.21.15.191:3300/0 conn(0x7f4fbc0c53c0 0x7f4fbc0c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:06:09.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.487+0000 7f4fc2afb640 1 --2- 172.21.15.186:0/2057333459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 0x7f4fbc0cb040 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009d90 tx=0x7f4fb402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=367042b9ae9932e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:09.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.487+0000 7f4fc12f8640 1 -- 172.21.15.186:0/2057333459 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4039070 con 0x7f4fbc0c57e0 2024-03-28T15:06:09.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.487+0000 7f4fc12f8640 1 -- 172.21.15.186:0/2057333459 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fb402cce0 con 0x7f4fbc0c57e0 2024-03-28T15:06:09.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.487+0000 7f4fc12f8640 1 -- 172.21.15.186:0/2057333459 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4034660 con 0x7f4fbc0c57e0 2024-03-28T15:06:09.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.488+0000 7f4fc455e640 1 -- 172.21.15.186:0/2057333459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 msgr2=0x7f4fbc0cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:09.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.488+0000 7f4fc455e640 1 --2- 172.21.15.186:0/2057333459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 0x7f4fbc0cb040 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009d90 tx=0x7f4fb402bf30 comp rx=0 tx=0).stop 2024-03-28T15:06:09.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.488+0000 7f4fc455e640 1 -- 172.21.15.186:0/2057333459 shutdown_connections 2024-03-28T15:06:09.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.488+0000 7f4fc455e640 1 --2- 172.21.15.186:0/2057333459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 0x7f4fbc0cb040 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:09.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.488+0000 7f4fc455e640 1 --2- 172.21.15.186:0/2057333459 >> v2:172.21.15.191:3300/0 conn(0x7f4fbc0c53c0 0x7f4fbc0c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:09.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.488+0000 7f4fc455e640 1 -- 172.21.15.186:0/2057333459 >> 172.21.15.186:0/2057333459 conn(0x7f4fbc0731f0 msgr2=0x7f4fbc073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:09.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.488+0000 7f4fc455e640 1 -- 172.21.15.186:0/2057333459 shutdown_connections 2024-03-28T15:06:09.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.488+0000 7f4fc455e640 1 -- 172.21.15.186:0/2057333459 wait complete. 2024-03-28T15:06:09.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.489+0000 7f4fc455e640 1 Processor -- start 2024-03-28T15:06:09.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.489+0000 7f4fc455e640 1 -- start start 2024-03-28T15:06:09.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc455e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc0c53c0 0x7f4fbc1537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:09.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc455e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 0x7f4fbc1f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:09.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc455e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fbc154bd0 0x7f4fbc1f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:09.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc455e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4fbc0cc390 con 0x7f4fbc154bd0 2024-03-28T15:06:09.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc455e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4fbc0c61d0 con 0x7f4fbc0c53c0 2024-03-28T15:06:09.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc455e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4fbc0c6c30 con 0x7f4fbc0c57e0 2024-03-28T15:06:09.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc2afb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fbc154bd0 0x7f4fbc1f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:09.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc2afb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fbc154bd0 0x7f4fbc1f3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39536/0 (socket says 172.21.15.186:39536) 2024-03-28T15:06:09.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc2afb640 1 -- 172.21.15.186:0/3375506434 learned_addr learned my addr 172.21.15.186:0/3375506434 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:09.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc22fa640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc0c53c0 0x7f4fbc1537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:09.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc1af9640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 0x7f4fbc1f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:09.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc2afb640 1 -- 172.21.15.186:0/3375506434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 msgr2=0x7f4fbc1f0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:09.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc2afb640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 0x7f4fbc1f0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:09.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc2afb640 1 -- 172.21.15.186:0/3375506434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc0c53c0 msgr2=0x7f4fbc1537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:09.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc2afb640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc0c53c0 0x7f4fbc1537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:09.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc2afb640 1 -- 172.21.15.186:0/3375506434 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fbc155330 con 0x7f4fbc154bd0 2024-03-28T15:06:09.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc22fa640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc0c53c0 0x7f4fbc1537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:06:09.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fc2afb640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fbc154bd0 0x7f4fbc1f3b60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009d90 tx=0x7f4fb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:09.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.490+0000 7f4fb2ffd640 1 -- 172.21.15.186:0/3375506434 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4039050 con 0x7f4fbc154bd0 2024-03-28T15:06:09.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.491+0000 7f4fb2ffd640 1 -- 172.21.15.186:0/3375506434 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fb4045070 con 0x7f4fbc154bd0 2024-03-28T15:06:09.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.491+0000 7f4fc455e640 1 -- 172.21.15.186:0/3375506434 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fbc155610 con 0x7f4fbc154bd0 2024-03-28T15:06:09.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.491+0000 7f4fb2ffd640 1 -- 172.21.15.186:0/3375506434 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4009450 con 0x7f4fbc154bd0 2024-03-28T15:06:09.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.491+0000 7f4fc455e640 1 -- 172.21.15.186:0/3375506434 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fbc155bd0 con 0x7f4fbc154bd0 2024-03-28T15:06:09.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.491+0000 7f4fc455e640 1 -- 172.21.15.186:0/3375506434 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fbc1f05f0 con 0x7f4fbc154bd0 2024-03-28T15:06:09.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.492+0000 7f4fb2ffd640 1 -- 172.21.15.186:0/3375506434 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4fb403e850 con 0x7f4fbc154bd0 2024-03-28T15:06:09.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.492+0000 7f4fb2ffd640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4fa006c530 0x7f4fa006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:09.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.492+0000 7f4fb2ffd640 1 -- 172.21.15.186:0/3375506434 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f4fb40ae650 con 0x7f4fbc154bd0 2024-03-28T15:06:09.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.492+0000 7f4fc22fa640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4fa006c530 0x7f4fa006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:09.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.493+0000 7f4fc22fa640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4fa006c530 0x7f4fa006e9f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8007fd0 tx=0x7f4fb8007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:09.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.497+0000 7f4fb2ffd640 1 -- 172.21.15.186:0/3375506434 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4fb407c5c0 con 0x7f4fbc154bd0 2024-03-28T15:06:09.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.710+0000 7f4fc455e640 1 -- 172.21.15.186:0/3375506434 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fbc069500 con 0x7f4fbc154bd0 2024-03-28T15:06:09.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.711+0000 7f4fb2ffd640 1 -- 172.21.15.186:0/3375506434 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6012 (secure 0 0 0) 0x7f4fb40ace20 con 0x7f4fbc154bd0 2024-03-28T15:06:09.711 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:09.711 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:06.202561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:06:09.711 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 20 2024-03-28T15:06:09.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 -- 172.21.15.186:0/3375506434 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4fa006c530 msgr2=0x7f4fa006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:09.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4fa006c530 0x7f4fa006e9f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8007fd0 tx=0x7f4fb8007e80 comp rx=0 tx=0).stop 2024-03-28T15:06:09.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 -- 172.21.15.186:0/3375506434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fbc154bd0 msgr2=0x7f4fbc1f3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:09.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fbc154bd0 0x7f4fbc1f3b60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009d90 tx=0x7f4fb4004a40 comp rx=0 tx=0).stop 2024-03-28T15:06:09.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 -- 172.21.15.186:0/3375506434 shutdown_connections 2024-03-28T15:06:09.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4fa006c530 0x7f4fa006e9f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:09.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fbc154bd0 0x7f4fbc1f3b60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:09.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fbc0c57e0 0x7f4fbc1f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:09.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 --2- 172.21.15.186:0/3375506434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc0c53c0 0x7f4fbc1537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:09.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 -- 172.21.15.186:0/3375506434 >> 172.21.15.186:0/3375506434 conn(0x7f4fbc0731f0 msgr2=0x7f4fbc1e3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:09.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 -- 172.21.15.186:0/3375506434 shutdown_connections 2024-03-28T15:06:09.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:09.713+0000 7f4fc455e640 1 -- 172.21.15.186:0/3375506434 wait complete. 2024-03-28T15:06:09.724 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:06:11.725 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:11.838 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.838+0000 7fbcdf913640 1 Processor -- start 2024-03-28T15:06:11.838 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.838+0000 7fbcdf913640 1 -- start start 2024-03-28T15:06:11.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.839+0000 7fbcdf913640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbcd80b1e30 0x7fbcd80712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:11.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.839+0000 7fbcdf913640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd806caa0 con 0x7fbcd8151990 2024-03-28T15:06:11.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.839+0000 7fbcdf913640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd806cd80 con 0x7fbcd80c2d00 2024-03-28T15:06:11.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.839+0000 7fbcdf913640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fbcd80c1da0 con 0x7fbcd80b1e30 2024-03-28T15:06:11.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.839+0000 7fbcdceae640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbcd80b1e30 0x7fbcd80712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:11.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.839+0000 7fbcdd6af640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fbcd8151990 0x7fbcd80c2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:40200/0 (socket says 172.21.15.186:40200) 2024-03-28T15:06:11.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.839+0000 7fbcdd6af640 1 -- 172.21.15.186:0/3367981716 learned_addr learned my addr 172.21.15.186:0/3367981716 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:06:11.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.839+0000 7fbcdceae640 1 -- 172.21.15.186:0/3367981716 >> v1:172.21.15.191:6789/0 conn(0x7fbcd80c2d00 legacy=0x7fbcd80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:11.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.839+0000 7fbcdceae640 1 -- 172.21.15.186:0/3367981716 >> v1:172.21.15.186:6789/0 conn(0x7fbcd8151990 legacy=0x7fbcd80c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:11.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.839+0000 7fbcdceae640 1 -- 172.21.15.186:0/3367981716 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd80c1340 con 0x7fbcd80b1e30 2024-03-28T15:06:11.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.839+0000 7fbccffff640 1 -- 172.21.15.186:0/3367981716 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 304533943 0 0) 0x7fbcd806caa0 con 0x7fbcd8151990 2024-03-28T15:06:11.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.840+0000 7fbcdceae640 1 --2- 172.21.15.186:0/3367981716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcd80b1e30 0x7fbcd80712b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80097a0 tx=0x7fbcc802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4551deb6eed92dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:11.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.840+0000 7fbccffff640 1 -- 172.21.15.186:0/3367981716 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc802f810 con 0x7fbcd80b1e30 2024-03-28T15:06:11.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.840+0000 7fbccffff640 1 -- 172.21.15.186:0/3367981716 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc802f9b0 con 0x7fbcd80b1e30 2024-03-28T15:06:11.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.840+0000 7fbccffff640 1 -- 172.21.15.186:0/3367981716 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc802fd80 con 0x7fbcd80b1e30 2024-03-28T15:06:11.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.841+0000 7fbcdf913640 1 -- 172.21.15.186:0/3367981716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcd80b1e30 msgr2=0x7fbcd80712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:11.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.841+0000 7fbcdf913640 1 --2- 172.21.15.186:0/3367981716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcd80b1e30 0x7fbcd80712b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80097a0 tx=0x7fbcc802b9f0 comp rx=0 tx=0).stop 2024-03-28T15:06:11.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.841+0000 7fbcdf913640 1 -- 172.21.15.186:0/3367981716 shutdown_connections 2024-03-28T15:06:11.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.841+0000 7fbcdf913640 1 --2- 172.21.15.186:0/3367981716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcd80b1e30 0x7fbcd80712b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:11.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.841+0000 7fbcdf913640 1 -- 172.21.15.186:0/3367981716 >> 172.21.15.186:0/3367981716 conn(0x7fbcd80c8a20 msgr2=0x7fbcd80c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:11.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.841+0000 7fbcdf913640 1 -- 172.21.15.186:0/3367981716 shutdown_connections 2024-03-28T15:06:11.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.841+0000 7fbcdf913640 1 -- 172.21.15.186:0/3367981716 wait complete. 2024-03-28T15:06:11.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.842+0000 7fbcdf913640 1 Processor -- start 2024-03-28T15:06:11.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.842+0000 7fbcdf913640 1 -- start start 2024-03-28T15:06:11.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdf913640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbcd80b1e30 0x7fbcd80caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:11.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdf913640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcd80c2d00 0x7fbcd81efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:11.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdf913640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbcd8151990 0x7fbcd81ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:11.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdf913640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd80c2940 con 0x7fbcd80c2d00 2024-03-28T15:06:11.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdf913640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd80c1900 con 0x7fbcd8151990 2024-03-28T15:06:11.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdf913640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd80c2ac0 con 0x7fbcd80b1e30 2024-03-28T15:06:11.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdceae640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcd80c2d00 0x7fbcd81efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:11.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdceae640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcd80c2d00 0x7fbcd81efc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39558/0 (socket says 172.21.15.186:39558) 2024-03-28T15:06:11.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdceae640 1 -- 172.21.15.186:0/3479607101 learned_addr learned my addr 172.21.15.186:0/3479607101 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:11.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcddeb0640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbcd8151990 0x7fbcd81ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:11.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdd6af640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbcd80b1e30 0x7fbcd80caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:11.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdceae640 1 -- 172.21.15.186:0/3479607101 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbcd80b1e30 msgr2=0x7fbcd80caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:11.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdceae640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbcd80b1e30 0x7fbcd80caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:11.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdceae640 1 -- 172.21.15.186:0/3479607101 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbcd8151990 msgr2=0x7fbcd81ef2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:11.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdceae640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbcd8151990 0x7fbcd81ef2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:11.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdceae640 1 -- 172.21.15.186:0/3479607101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd80cb5c0 con 0x7fbcd80c2d00 2024-03-28T15:06:11.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcddeb0640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbcd8151990 0x7fbcd81ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:06:11.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdceae640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcd80c2d00 0x7fbcd81efc00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbcc802c000 tx=0x7fbcc8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:11.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbccdffb640 1 -- 172.21.15.186:0/3479607101 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8046070 con 0x7fbcd80c2d00 2024-03-28T15:06:11.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbccdffb640 1 -- 172.21.15.186:0/3479607101 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc8037c60 con 0x7fbcd80c2d00 2024-03-28T15:06:11.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbcdf913640 1 -- 172.21.15.186:0/3479607101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd80cb860 con 0x7fbcd80c2d00 2024-03-28T15:06:11.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.843+0000 7fbccdffb640 1 -- 172.21.15.186:0/3479607101 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc802fe60 con 0x7fbcd80c2d00 2024-03-28T15:06:11.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.844+0000 7fbcdf913640 1 -- 172.21.15.186:0/3479607101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd81eeac0 con 0x7fbcd80c2d00 2024-03-28T15:06:11.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.845+0000 7fbcdf913640 1 -- 172.21.15.186:0/3479607101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd8069500 con 0x7fbcd80c2d00 2024-03-28T15:06:11.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.845+0000 7fbccdffb640 1 -- 172.21.15.186:0/3479607101 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbcc8049070 con 0x7fbcd80c2d00 2024-03-28T15:06:11.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.845+0000 7fbccdffb640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbcbc06c530 0x7fbcbc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:11.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.846+0000 7fbccdffb640 1 -- 172.21.15.186:0/3479607101 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7fbcc80ad500 con 0x7fbcd80c2d00 2024-03-28T15:06:11.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.846+0000 7fbcdd6af640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbcbc06c530 0x7fbcbc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:11.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.846+0000 7fbcdd6af640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbcbc06c530 0x7fbcbc06e9f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4000f60 tx=0x7fbcd4007b90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:11.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:11.850+0000 7fbccdffb640 1 -- 172.21.15.186:0/3479607101 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fbcc8033030 con 0x7fbcd80c2d00 2024-03-28T15:06:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.064+0000 7fbcdf913640 1 -- 172.21.15.186:0/3479607101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcd80b16d0 con 0x7fbcd80c2d00 2024-03-28T15:06:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.064+0000 7fbccdffb640 1 -- 172.21.15.186:0/3479607101 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6012 (secure 0 0 0) 0x7fbcc807bac0 con 0x7fbcd80c2d00 2024-03-28T15:06:12.064 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:12.065 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:06.202561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:06:12.065 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 20 2024-03-28T15:06:12.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 -- 172.21.15.186:0/3479607101 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbcbc06c530 msgr2=0x7fbcbc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:12.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbcbc06c530 0x7fbcbc06e9f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4000f60 tx=0x7fbcd4007b90 comp rx=0 tx=0).stop 2024-03-28T15:06:12.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 -- 172.21.15.186:0/3479607101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcd80c2d00 msgr2=0x7fbcd81efc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:12.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcd80c2d00 0x7fbcd81efc00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbcc802c000 tx=0x7fbcc8004730 comp rx=0 tx=0).stop 2024-03-28T15:06:12.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 -- 172.21.15.186:0/3479607101 shutdown_connections 2024-03-28T15:06:12.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbcbc06c530 0x7fbcbc06e9f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:12.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbcd8151990 0x7fbcd81ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:12.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcd80c2d00 0x7fbcd81efc00 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:12.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 --2- 172.21.15.186:0/3479607101 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbcd80b1e30 0x7fbcd80caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:12.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 -- 172.21.15.186:0/3479607101 >> 172.21.15.186:0/3479607101 conn(0x7fbcd80c8a20 msgr2=0x7fbcd814ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:12.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 -- 172.21.15.186:0/3479607101 shutdown_connections 2024-03-28T15:06:12.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:12.066+0000 7fbcdf913640 1 -- 172.21.15.186:0/3479607101 wait complete. 2024-03-28T15:06:12.077 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:06:14.078 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:14.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.191+0000 7f41b965c640 1 Processor -- start 2024-03-28T15:06:14.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.191+0000 7f41b965c640 1 -- start start 2024-03-28T15:06:14.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b965c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41b4154e40 0x7f41b40c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:14.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b965c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f41b415e080 0x7f41b40c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:14.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b965c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41b40c8a90 0x7f41b406f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:14.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b965c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f41b40caa10 con 0x7f41b4154e40 2024-03-28T15:06:14.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b965c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f41b40c4810 con 0x7f41b415e080 2024-03-28T15:06:14.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b965c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f41b40b2ab0 con 0x7f41b40c8a90 2024-03-28T15:06:14.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b2ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41b4154e40 0x7f41b40c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:14.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b2ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41b4154e40 0x7f41b40c42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39572/0 (socket says 172.21.15.186:39572) 2024-03-28T15:06:14.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b2ffd640 1 -- 172.21.15.186:0/3826554659 learned_addr learned my addr 172.21.15.186:0/3826554659 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:14.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b37fe640 1 --2- 172.21.15.186:0/3826554659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41b40c8a90 0x7f41b406f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:14.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b27fc640 1 --2- 172.21.15.186:0/3826554659 >> v2:172.21.15.191:3300/0 conn(0x7f41b415e080 0x7f41b40c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:14.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b2ffd640 1 -- 172.21.15.186:0/3826554659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41b40c8a90 msgr2=0x7f41b406f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:14.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b2ffd640 1 --2- 172.21.15.186:0/3826554659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41b40c8a90 0x7f41b406f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b2ffd640 1 -- 172.21.15.186:0/3826554659 >> v2:172.21.15.191:3300/0 conn(0x7f41b415e080 msgr2=0x7f41b40c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:14.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b2ffd640 1 --2- 172.21.15.186:0/3826554659 >> v2:172.21.15.191:3300/0 conn(0x7f41b415e080 0x7f41b40c30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.192+0000 7f41b2ffd640 1 -- 172.21.15.186:0/3826554659 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b40b2d50 con 0x7f41b4154e40 2024-03-28T15:06:14.195 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.193+0000 7f41b2ffd640 1 --2- 172.21.15.186:0/3826554659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41b4154e40 0x7f41b40c42d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f41b406e690 tx=0x7f41a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5673000d56464bad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.193+0000 7f41b1ffb640 1 -- 172.21.15.186:0/3826554659 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4035020 con 0x7f41b4154e40 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.193+0000 7f41b1ffb640 1 -- 172.21.15.186:0/3826554659 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a4031640 con 0x7f41b4154e40 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.193+0000 7f41b1ffb640 1 -- 172.21.15.186:0/3826554659 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a402d040 con 0x7f41b4154e40 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.194+0000 7f41b965c640 1 -- 172.21.15.186:0/3826554659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41b4154e40 msgr2=0x7f41b40c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.194+0000 7f41b965c640 1 --2- 172.21.15.186:0/3826554659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41b4154e40 0x7f41b40c42d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f41b406e690 tx=0x7f41a402b8f0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.194+0000 7f41b965c640 1 -- 172.21.15.186:0/3826554659 shutdown_connections 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.194+0000 7f41b965c640 1 --2- 172.21.15.186:0/3826554659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41b40c8a90 0x7f41b406f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.194+0000 7f41b965c640 1 --2- 172.21.15.186:0/3826554659 >> v2:172.21.15.191:3300/0 conn(0x7f41b415e080 0x7f41b40c30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.194+0000 7f41b965c640 1 --2- 172.21.15.186:0/3826554659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41b4154e40 0x7f41b40c42d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.194+0000 7f41b965c640 1 -- 172.21.15.186:0/3826554659 >> 172.21.15.186:0/3826554659 conn(0x7f41b4067280 msgr2=0x7f41b4067660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.194+0000 7f41b965c640 1 -- 172.21.15.186:0/3826554659 shutdown_connections 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.194+0000 7f41b965c640 1 -- 172.21.15.186:0/3826554659 wait complete. 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.195+0000 7f41b965c640 1 Processor -- start 2024-03-28T15:06:14.196 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.195+0000 7f41b965c640 1 -- start start 2024-03-28T15:06:14.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.196+0000 7f41b965c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41b40c8a90 0x7f41b41e7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:14.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.196+0000 7f41b965c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41b4154e40 0x7f41b406db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:14.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.196+0000 7f41b965c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41b415e080 0x7f41b41f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.196+0000 7f41b965c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41b40b35d0 con 0x7f41b4154e40 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.196+0000 7f41b965c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f41b40b32f0 con 0x7f41b40c8a90 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.196+0000 7f41b965c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f41b40c3ad0 con 0x7f41b415e080 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.196+0000 7f41b27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41b4154e40 0x7f41b406db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.196+0000 7f41b27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41b4154e40 0x7f41b406db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39580/0 (socket says 172.21.15.186:39580) 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.196+0000 7f41b27fc640 1 -- 172.21.15.186:0/1549115786 learned_addr learned my addr 172.21.15.186:0/1549115786 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.196+0000 7f41b2ffd640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41b40c8a90 0x7f41b41e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.196+0000 7f41b37fe640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41b415e080 0x7f41b41f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.197+0000 7f41b27fc640 1 -- 172.21.15.186:0/1549115786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41b415e080 msgr2=0x7f41b41f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.197+0000 7f41b27fc640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41b415e080 0x7f41b41f06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.197+0000 7f41b27fc640 1 -- 172.21.15.186:0/1549115786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41b40c8a90 msgr2=0x7f41b41e7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.197+0000 7f41b27fc640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41b40c8a90 0x7f41b41e7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.197+0000 7f41b27fc640 1 -- 172.21.15.186:0/1549115786 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b41f3c80 con 0x7f41b4154e40 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.197+0000 7f41b27fc640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41b4154e40 0x7f41b406db10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f41a800d5f0 tx=0x7f41a800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:14.199 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.197+0000 7f419b7fe640 1 -- 172.21.15.186:0/1549115786 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a80125f0 con 0x7f41b4154e40 2024-03-28T15:06:14.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.197+0000 7f419b7fe640 1 -- 172.21.15.186:0/1549115786 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a8019070 con 0x7f41b4154e40 2024-03-28T15:06:14.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.197+0000 7f41b965c640 1 -- 172.21.15.186:0/1549115786 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b41f3f20 con 0x7f41b4154e40 2024-03-28T15:06:14.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.197+0000 7f419b7fe640 1 -- 172.21.15.186:0/1549115786 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a80129d0 con 0x7f41b4154e40 2024-03-28T15:06:14.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.197+0000 7f41b965c640 1 -- 172.21.15.186:0/1549115786 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b41f44e0 con 0x7f41b4154e40 2024-03-28T15:06:14.200 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.198+0000 7f41b965c640 1 -- 172.21.15.186:0/1549115786 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b40690d0 con 0x7f41b4154e40 2024-03-28T15:06:14.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.198+0000 7f419b7fe640 1 -- 172.21.15.186:0/1549115786 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f41a8012b70 con 0x7f41b4154e40 2024-03-28T15:06:14.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.199+0000 7f419b7fe640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f41a006c560 0x7f41a006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:14.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.199+0000 7f419b7fe640 1 -- 172.21.15.186:0/1549115786 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f41a808ba00 con 0x7f41b4154e40 2024-03-28T15:06:14.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.199+0000 7f41b2ffd640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f41a006c560 0x7f41a006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:14.201 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.199+0000 7f41b2ffd640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f41a006c560 0x7f41a006ea20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f41a4036000 tx=0x7f41a403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:14.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.203+0000 7f419b7fe640 1 -- 172.21.15.186:0/1549115786 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f41b40690d0 con 0x7f41b4154e40 2024-03-28T15:06:14.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.416+0000 7f41b965c640 1 -- 172.21.15.186:0/1549115786 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41b4069500 con 0x7f41b4154e40 2024-03-28T15:06:14.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.417+0000 7f419b7fe640 1 -- 172.21.15.186:0/1549115786 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6016 (secure 0 0 0) 0x7f41a80598e0 con 0x7f41b4154e40 2024-03-28T15:06:14.418 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:14.418 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:13.303665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:06:14.421 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 21 2024-03-28T15:06:14.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 -- 172.21.15.186:0/1549115786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f41a006c560 msgr2=0x7f41a006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:14.422 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f41a006c560 0x7f41a006ea20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f41a4036000 tx=0x7f41a403c040 comp rx=0 tx=0).stop 2024-03-28T15:06:14.422 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 -- 172.21.15.186:0/1549115786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41b4154e40 msgr2=0x7f41b406db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:14.422 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41b4154e40 0x7f41b406db10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f41a800d5f0 tx=0x7f41a800dab0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.422 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 -- 172.21.15.186:0/1549115786 shutdown_connections 2024-03-28T15:06:14.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f41a006c560 0x7f41a006ea20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41b415e080 0x7f41b41f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41b4154e40 0x7f41b406db10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 --2- 172.21.15.186:0/1549115786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41b40c8a90 0x7f41b41e7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:14.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 -- 172.21.15.186:0/1549115786 >> 172.21.15.186:0/1549115786 conn(0x7f41b4067280 msgr2=0x7f41b415c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:14.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 -- 172.21.15.186:0/1549115786 shutdown_connections 2024-03-28T15:06:14.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:14.419+0000 7f41b965c640 1 -- 172.21.15.186:0/1549115786 wait complete. 2024-03-28T15:06:14.430 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:06:16.431 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:16.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.543+0000 7faf63f48640 1 Processor -- start 2024-03-28T15:06:16.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.543+0000 7faf63f48640 1 -- start start 2024-03-28T15:06:16.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.544+0000 7faf63f48640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faf5c0b1e30 0x7faf5c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:16.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.544+0000 7faf63f48640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf5c06caa0 con 0x7faf5c0c2d00 2024-03-28T15:06:16.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.544+0000 7faf63f48640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf5c06cd80 con 0x7faf5c151990 2024-03-28T15:06:16.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.544+0000 7faf63f48640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7faf5c0c1da0 con 0x7faf5c0b1e30 2024-03-28T15:06:16.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.544+0000 7faf624e5640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7faf5c0c2d00 0x7faf5c0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:40210/0 (socket says 172.21.15.186:40210) 2024-03-28T15:06:16.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.544+0000 7faf624e5640 1 -- 172.21.15.186:0/4265944190 learned_addr learned my addr 172.21.15.186:0/4265944190 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:06:16.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.544+0000 7faf614e3640 1 --2- 172.21.15.186:0/4265944190 >> v2:172.21.15.191:3300/0 conn(0x7faf5c0b1e30 0x7faf5c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:16.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.545+0000 7faf60ce2640 1 -- 172.21.15.186:0/4265944190 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 358302874 0 0) 0x7faf5c06caa0 con 0x7faf5c0c2d00 2024-03-28T15:06:16.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.545+0000 7faf60ce2640 1 -- 172.21.15.186:0/4265944190 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf48003520 con 0x7faf5c0c2d00 2024-03-28T15:06:16.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.545+0000 7faf60ce2640 1 -- 172.21.15.186:0/4265944190 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4265020982 0 0) 0x7faf5c06cd80 con 0x7faf5c151990 2024-03-28T15:06:16.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.545+0000 7faf60ce2640 1 -- 172.21.15.186:0/4265944190 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf5c06caa0 con 0x7faf5c151990 2024-03-28T15:06:16.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.545+0000 7faf614e3640 1 -- 172.21.15.186:0/4265944190 >> v1:172.21.15.191:6789/0 conn(0x7faf5c151990 legacy=0x7faf5c0c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:16.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.545+0000 7faf614e3640 1 -- 172.21.15.186:0/4265944190 >> v1:172.21.15.186:6789/0 conn(0x7faf5c0c2d00 legacy=0x7faf5c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:16.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.545+0000 7faf614e3640 1 -- 172.21.15.186:0/4265944190 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5c0c1340 con 0x7faf5c0b1e30 2024-03-28T15:06:16.547 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.546+0000 7faf614e3640 1 --2- 172.21.15.186:0/4265944190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5c0b1e30 0x7faf5c0712b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0097a0 tx=0x7faf4c02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec28f1deb79e8a31 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:16.547 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.546+0000 7faf60ce2640 1 -- 172.21.15.186:0/4265944190 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c02f810 con 0x7faf5c0b1e30 2024-03-28T15:06:16.547 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.546+0000 7faf60ce2640 1 -- 172.21.15.186:0/4265944190 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf4c02f9b0 con 0x7faf5c0b1e30 2024-03-28T15:06:16.547 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.546+0000 7faf60ce2640 1 -- 172.21.15.186:0/4265944190 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c02fd60 con 0x7faf5c0b1e30 2024-03-28T15:06:16.548 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.546+0000 7faf63f48640 1 -- 172.21.15.186:0/4265944190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5c0b1e30 msgr2=0x7faf5c0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:16.548 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.546+0000 7faf63f48640 1 --2- 172.21.15.186:0/4265944190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5c0b1e30 0x7faf5c0712b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0097a0 tx=0x7faf4c02b9f0 comp rx=0 tx=0).stop 2024-03-28T15:06:16.548 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.547+0000 7faf63f48640 1 -- 172.21.15.186:0/4265944190 shutdown_connections 2024-03-28T15:06:16.548 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.547+0000 7faf63f48640 1 --2- 172.21.15.186:0/4265944190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5c0b1e30 0x7faf5c0712b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:16.548 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.547+0000 7faf63f48640 1 -- 172.21.15.186:0/4265944190 >> 172.21.15.186:0/4265944190 conn(0x7faf5c0c8a20 msgr2=0x7faf5c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:16.548 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.547+0000 7faf63f48640 1 -- 172.21.15.186:0/4265944190 shutdown_connections 2024-03-28T15:06:16.548 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.547+0000 7faf63f48640 1 -- 172.21.15.186:0/4265944190 wait complete. 2024-03-28T15:06:16.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.548+0000 7faf63f48640 1 Processor -- start 2024-03-28T15:06:16.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.548+0000 7faf63f48640 1 -- start start 2024-03-28T15:06:16.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.548+0000 7faf63f48640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf5c0b1e30 0x7faf5c0cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:16.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.548+0000 7faf63f48640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5c0c2d00 0x7faf5c0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.548+0000 7faf63f48640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c151990 0x7faf5c14d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.548+0000 7faf63f48640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c0c2940 con 0x7faf5c151990 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.548+0000 7faf63f48640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c0c1900 con 0x7faf5c0c2d00 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.548+0000 7faf63f48640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7faf5c0c2ac0 con 0x7faf5c0b1e30 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf624e5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c151990 0x7faf5c14d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf624e5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c151990 0x7faf5c14d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39586/0 (socket says 172.21.15.186:39586) 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf624e5640 1 -- 172.21.15.186:0/3949912846 learned_addr learned my addr 172.21.15.186:0/3949912846 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf614e3640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5c0c2d00 0x7faf5c0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf61ce4640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf5c0b1e30 0x7faf5c0cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf624e5640 1 -- 172.21.15.186:0/3949912846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf5c0b1e30 msgr2=0x7faf5c0cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf624e5640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf5c0b1e30 0x7faf5c0cb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf624e5640 1 -- 172.21.15.186:0/3949912846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5c0c2d00 msgr2=0x7faf5c0ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf624e5640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5c0c2d00 0x7faf5c0ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf624e5640 1 -- 172.21.15.186:0/3949912846 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5c0c6380 con 0x7faf5c151990 2024-03-28T15:06:16.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf624e5640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c151990 0x7faf5c14d1e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7faf5400c390 tx=0x7faf5400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:16.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf527fc640 1 -- 172.21.15.186:0/3949912846 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf54017070 con 0x7faf5c151990 2024-03-28T15:06:16.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf527fc640 1 -- 172.21.15.186:0/3949912846 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf54005ce0 con 0x7faf5c151990 2024-03-28T15:06:16.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf63f48640 1 -- 172.21.15.186:0/3949912846 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf5c0c55a0 con 0x7faf5c151990 2024-03-28T15:06:16.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf527fc640 1 -- 172.21.15.186:0/3949912846 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf54007310 con 0x7faf5c151990 2024-03-28T15:06:16.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.549+0000 7faf63f48640 1 -- 172.21.15.186:0/3949912846 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf5c0c5b40 con 0x7faf5c151990 2024-03-28T15:06:16.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.550+0000 7faf63f48640 1 -- 172.21.15.186:0/3949912846 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf5c067740 con 0x7faf5c151990 2024-03-28T15:06:16.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.551+0000 7faf527fc640 1 -- 172.21.15.186:0/3949912846 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7faf5401a070 con 0x7faf5c151990 2024-03-28T15:06:16.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.551+0000 7faf527fc640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faf4006c530 0x7faf4006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:16.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.551+0000 7faf61ce4640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faf4006c530 0x7faf4006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:16.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.551+0000 7faf527fc640 1 -- 172.21.15.186:0/3949912846 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7faf5408ac30 con 0x7faf5c151990 2024-03-28T15:06:16.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.552+0000 7faf61ce4640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faf4006c530 0x7faf4006e9f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7faf58001ea0 tx=0x7faf58009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:16.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.556+0000 7faf527fc640 1 -- 172.21.15.186:0/3949912846 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7faf5c067740 con 0x7faf5c151990 2024-03-28T15:06:16.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.768+0000 7faf63f48640 1 -- 172.21.15.186:0/3949912846 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf5c0c7db0 con 0x7faf5c151990 2024-03-28T15:06:16.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.769+0000 7faf527fc640 1 -- 172.21.15.186:0/3949912846 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6016 (secure 0 0 0) 0x7faf5c0c7db0 con 0x7faf5c151990 2024-03-28T15:06:16.769 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:16.769 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:13.303665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:06:16.770 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 21 2024-03-28T15:06:16.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 -- 172.21.15.186:0/3949912846 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faf4006c530 msgr2=0x7faf4006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:16.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faf4006c530 0x7faf4006e9f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7faf58001ea0 tx=0x7faf58009040 comp rx=0 tx=0).stop 2024-03-28T15:06:16.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 -- 172.21.15.186:0/3949912846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c151990 msgr2=0x7faf5c14d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:16.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c151990 0x7faf5c14d1e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7faf5400c390 tx=0x7faf5400c850 comp rx=0 tx=0).stop 2024-03-28T15:06:16.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 -- 172.21.15.186:0/3949912846 shutdown_connections 2024-03-28T15:06:16.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7faf4006c530 0x7faf4006e9f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:16.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf5c151990 0x7faf5c14d1e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:16.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5c0c2d00 0x7faf5c0ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:16.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 --2- 172.21.15.186:0/3949912846 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf5c0b1e30 0x7faf5c0cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:16.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 -- 172.21.15.186:0/3949912846 >> 172.21.15.186:0/3949912846 conn(0x7faf5c0c8a20 msgr2=0x7faf5c0732c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:16.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 -- 172.21.15.186:0/3949912846 shutdown_connections 2024-03-28T15:06:16.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:16.771+0000 7faf63f48640 1 -- 172.21.15.186:0/3949912846 wait complete. 2024-03-28T15:06:16.783 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:06:18.784 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.896+0000 7fc66b48d640 1 Processor -- start 2024-03-28T15:06:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.896+0000 7fc66b48d640 1 -- start start 2024-03-28T15:06:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.897+0000 7fc66b48d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc664151990 0x7fc6640c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.897+0000 7fc66b48d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6640c2d00 0x7fc6640c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.897+0000 7fc66b48d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc66406caa0 con 0x7fc6640b1e30 2024-03-28T15:06:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.897+0000 7fc66b48d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc6640c1da0 con 0x7fc664151990 2024-03-28T15:06:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.897+0000 7fc66b48d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc66406cd80 con 0x7fc6640c2d00 2024-03-28T15:06:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.897+0000 7fc669229640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc664151990 0x7fc6640c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:18.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.897+0000 7fc669229640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc664151990 0x7fc6640c2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:51804/0 (socket says 172.21.15.186:51804) 2024-03-28T15:06:18.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.897+0000 7fc669229640 1 -- 172.21.15.186:0/2838940215 learned_addr learned my addr 172.21.15.186:0/2838940215 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:18.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.897+0000 7fc669a2a640 1 --2- 172.21.15.186:0/2838940215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6640c2d00 0x7fc6640c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:18.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.897+0000 7fc669229640 1 -- 172.21.15.186:0/2838940215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6640c2d00 msgr2=0x7fc6640c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:18.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.897+0000 7fc669229640 1 --2- 172.21.15.186:0/2838940215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6640c2d00 0x7fc6640c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:18.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.898+0000 7fc669229640 1 -- 172.21.15.186:0/2838940215 >> v1:172.21.15.191:6789/0 conn(0x7fc6640b1e30 legacy=0x7fc6640712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:18.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.898+0000 7fc669229640 1 -- 172.21.15.186:0/2838940215 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6640c1340 con 0x7fc664151990 2024-03-28T15:06:18.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.898+0000 7fc669229640 1 --2- 172.21.15.186:0/2838940215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc664151990 0x7fc6640c2230 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc660009e50 tx=0x7fc66002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf3c0aef01303c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:18.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.898+0000 7fc65bfff640 1 -- 172.21.15.186:0/2838940215 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66002cb90 con 0x7fc664151990 2024-03-28T15:06:18.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.898+0000 7fc65bfff640 1 -- 172.21.15.186:0/2838940215 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc66002cd30 con 0x7fc664151990 2024-03-28T15:06:18.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.898+0000 7fc65bfff640 1 -- 172.21.15.186:0/2838940215 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc660034610 con 0x7fc664151990 2024-03-28T15:06:18.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.899+0000 7fc66b48d640 1 -- 172.21.15.186:0/2838940215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc664151990 msgr2=0x7fc6640c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.899+0000 7fc66b48d640 1 --2- 172.21.15.186:0/2838940215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc664151990 0x7fc6640c2230 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc660009e50 tx=0x7fc66002bff0 comp rx=0 tx=0).stop 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.899+0000 7fc66b48d640 1 -- 172.21.15.186:0/2838940215 shutdown_connections 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.899+0000 7fc66b48d640 1 --2- 172.21.15.186:0/2838940215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6640c2d00 0x7fc6640c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.899+0000 7fc66b48d640 1 --2- 172.21.15.186:0/2838940215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc664151990 0x7fc6640c2230 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.899+0000 7fc66b48d640 1 -- 172.21.15.186:0/2838940215 >> 172.21.15.186:0/2838940215 conn(0x7fc6640c8a20 msgr2=0x7fc6640c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.899+0000 7fc66b48d640 1 -- 172.21.15.186:0/2838940215 shutdown_connections 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.899+0000 7fc66b48d640 1 -- 172.21.15.186:0/2838940215 wait complete. 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.900+0000 7fc66b48d640 1 Processor -- start 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.900+0000 7fc66b48d640 1 -- start start 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc66b48d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6640b1e30 0x7fc66414c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc66b48d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6640c2d00 0x7fc66406d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc66b48d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc664151990 0x7fc66406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc66b48d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc664073dd0 con 0x7fc664151990 2024-03-28T15:06:18.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc66b48d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc6640c1900 con 0x7fc6640c2d00 2024-03-28T15:06:18.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc66b48d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc664073f50 con 0x7fc6640b1e30 2024-03-28T15:06:18.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc669a2a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc664151990 0x7fc66406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:18.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc669a2a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc664151990 0x7fc66406dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:51820/0 (socket says 172.21.15.186:51820) 2024-03-28T15:06:18.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc669a2a640 1 -- 172.21.15.186:0/2821535754 learned_addr learned my addr 172.21.15.186:0/2821535754 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:18.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc669229640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6640b1e30 0x7fc66414c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:18.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc668a28640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6640c2d00 0x7fc66406d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:18.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc669a2a640 1 -- 172.21.15.186:0/2821535754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6640b1e30 msgr2=0x7fc66414c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:18.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc669a2a640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6640b1e30 0x7fc66414c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:18.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc669a2a640 1 -- 172.21.15.186:0/2821535754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6640c2d00 msgr2=0x7fc66406d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:18.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc669a2a640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6640c2d00 0x7fc66406d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:18.906 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.901+0000 7fc669a2a640 1 -- 172.21.15.186:0/2821535754 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6641f1dd0 con 0x7fc664151990 2024-03-28T15:06:18.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.902+0000 7fc669a2a640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc664151990 0x7fc66406dc60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00e820 tx=0x7fc65c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:18.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.902+0000 7fc659ffb640 1 -- 172.21.15.186:0/2821535754 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c017070 con 0x7fc664151990 2024-03-28T15:06:18.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.902+0000 7fc659ffb640 1 -- 172.21.15.186:0/2821535754 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc65c00ee70 con 0x7fc664151990 2024-03-28T15:06:18.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.902+0000 7fc66b48d640 1 -- 172.21.15.186:0/2821535754 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6641f20b0 con 0x7fc664151990 2024-03-28T15:06:18.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.902+0000 7fc659ffb640 1 -- 172.21.15.186:0/2821535754 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c00cb60 con 0x7fc664151990 2024-03-28T15:06:18.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.902+0000 7fc66b48d640 1 -- 172.21.15.186:0/2821535754 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6641f2670 con 0x7fc664151990 2024-03-28T15:06:18.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.903+0000 7fc66b48d640 1 -- 172.21.15.186:0/2821535754 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc664067740 con 0x7fc664151990 2024-03-28T15:06:18.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.904+0000 7fc659ffb640 1 -- 172.21.15.186:0/2821535754 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc65c01a070 con 0x7fc664151990 2024-03-28T15:06:18.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.904+0000 7fc659ffb640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc64806c530 0x7fc64806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:18.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.904+0000 7fc659ffb640 1 -- 172.21.15.186:0/2821535754 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7fc65c08b770 con 0x7fc664151990 2024-03-28T15:06:18.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.904+0000 7fc669229640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc64806c530 0x7fc64806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:18.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.905+0000 7fc669229640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc64806c530 0x7fc64806e9f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc660035000 tx=0x7fc660030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:18.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:18.909+0000 7fc659ffb640 1 -- 172.21.15.186:0/2821535754 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc664067740 con 0x7fc664151990 2024-03-28T15:06:19.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.120+0000 7fc66b48d640 1 -- 172.21.15.186:0/2821535754 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc664069500 con 0x7fc664151990 2024-03-28T15:06:19.121 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.121+0000 7fc659ffb640 1 -- 172.21.15.186:0/2821535754 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6013 (secure 0 0 0) 0x7fc65c059530 con 0x7fc664151990 2024-03-28T15:06:19.122 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:19.122 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:18.356264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:06:19.122 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 22 2024-03-28T15:06:19.125 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 -- 172.21.15.186:0/2821535754 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc64806c530 msgr2=0x7fc64806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:19.125 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc64806c530 0x7fc64806e9f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc660035000 tx=0x7fc660030000 comp rx=0 tx=0).stop 2024-03-28T15:06:19.125 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 -- 172.21.15.186:0/2821535754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc664151990 msgr2=0x7fc66406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:19.125 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc664151990 0x7fc66406dc60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00e820 tx=0x7fc65c00ec20 comp rx=0 tx=0).stop 2024-03-28T15:06:19.126 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 -- 172.21.15.186:0/2821535754 shutdown_connections 2024-03-28T15:06:19.126 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc64806c530 0x7fc64806e9f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:19.126 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc664151990 0x7fc66406dc60 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:19.126 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6640c2d00 0x7fc66406d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:19.126 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 --2- 172.21.15.186:0/2821535754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6640b1e30 0x7fc66414c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:19.126 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 -- 172.21.15.186:0/2821535754 >> 172.21.15.186:0/2821535754 conn(0x7fc6640c8a20 msgr2=0x7fc6640c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:19.126 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 -- 172.21.15.186:0/2821535754 shutdown_connections 2024-03-28T15:06:19.126 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:19.123+0000 7fc66b48d640 1 -- 172.21.15.186:0/2821535754 wait complete. 2024-03-28T15:06:19.135 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T15:01:31.425299+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4790": { "addr": "172.21.15.191:6834/1992706523", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 1992706523, "type": "v2" }, { "addr": "172.21.15.191:6834", "nonce": 1992706523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4790, "incarnation": 20, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T15:06:18.356264+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4790 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/1278716016", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6833", "nonce": 1278716016, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1278716016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6835/2068761001", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2068761001, "type": "v2" }, { "addr": "172.21.15.186:6835", "nonce": 2068761001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2344737463", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6837", "nonce": 2344737463, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2344737463, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4284, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/1548903381", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 1548903381, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 1548903381, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T15:06:21.136 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:21.249 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.249+0000 7eff42081640 1 Processor -- start 2024-03-28T15:06:21.249 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.249+0000 7eff42081640 1 -- start start 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.249+0000 7eff42081640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff3c154bd0 0x7eff3c15a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.249+0000 7eff42081640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3c0c6e00 con 0x7eff3c0c53c0 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.249+0000 7eff42081640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3c15b080 con 0x7eff3c0c57e0 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.249+0000 7eff42081640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7eff3c0c5ca0 con 0x7eff3c154bd0 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3affd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7eff3c0c53c0 0x7eff3c0c66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:36510/0 (socket says 172.21.15.186:36510) 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3affd640 1 -- 172.21.15.186:0/1013887357 learned_addr learned my addr 172.21.15.186:0/1013887357 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3b7fe640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff3c154bd0 0x7eff3c15a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938465472 0 0) 0x7eff3c0c6e00 con 0x7eff3c0c53c0 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff24003520 con 0x7eff3c0c53c0 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 460700398 0 0) 0x7eff24003520 con 0x7eff3c0c53c0 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff3c154bd0 msgr2=0x7eff3c15a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff3c154bd0 0x7eff3c15a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 >> v1:172.21.15.191:6789/0 conn(0x7eff3c0c57e0 legacy=0x7eff3c0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:21.253 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3c15b360 con 0x7eff3c0c53c0 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7eff30001670 con 0x7eff3c0c53c0 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 >> v1:172.21.15.186:6789/0 conn(0x7eff3c0c53c0 legacy=0x7eff3c0c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff240039e0 0x7eff24003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff3c154bd0 0x7eff24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff24004d40 0x7eff240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c0c5ca0 con 0x7eff240039e0 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7eff2400bb00 con 0x7eff24004d40 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7eff2400bd00 con 0x7eff3c154bd0 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3b7fe640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff240039e0 0x7eff24003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.250+0000 7eff3affd640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff3c154bd0 0x7eff24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.251+0000 7eff3bfff640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff24004d40 0x7eff240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:21.254 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.251+0000 7eff3b7fe640 1 -- 172.21.15.186:0/1013887357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff3c154bd0 msgr2=0x7eff24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.251+0000 7eff3b7fe640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff3c154bd0 0x7eff24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.251+0000 7eff3b7fe640 1 -- 172.21.15.186:0/1013887357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff24004d40 msgr2=0x7eff240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.251+0000 7eff3b7fe640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff24004d40 0x7eff240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.251+0000 7eff3b7fe640 1 -- 172.21.15.186:0/1013887357 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff24003520 con 0x7eff240039e0 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.251+0000 7eff3b7fe640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff240039e0 0x7eff24003e00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009a90 tx=0x7eff24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75e18c287e16897 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.252+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c039070 con 0x7eff240039e0 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.252+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2c02c680 con 0x7eff240039e0 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.252+0000 7eff3a7fc640 1 -- 172.21.15.186:0/1013887357 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c009690 con 0x7eff240039e0 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.252+0000 7eff42081640 1 -- 172.21.15.186:0/1013887357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff240039e0 msgr2=0x7eff24003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.252+0000 7eff42081640 1 --2- 172.21.15.186:0/1013887357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff240039e0 0x7eff24003e00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009a90 tx=0x7eff24005830 comp rx=0 tx=0).stop 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.252+0000 7eff3bfff640 1 -- 172.21.15.186:0/1013887357 reap_dead start 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.252+0000 7eff42081640 1 -- 172.21.15.186:0/1013887357 shutdown_connections 2024-03-28T15:06:21.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.252+0000 7eff42081640 1 -- 172.21.15.186:0/1013887357 >> 172.21.15.186:0/1013887357 conn(0x7eff3c0731f0 msgr2=0x7eff3c073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:21.256 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.252+0000 7eff42081640 1 -- 172.21.15.186:0/1013887357 shutdown_connections 2024-03-28T15:06:21.256 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.252+0000 7eff42081640 1 -- 172.21.15.186:0/1013887357 wait complete. 2024-03-28T15:06:21.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.253+0000 7eff42081640 1 Processor -- start 2024-03-28T15:06:21.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.254+0000 7eff42081640 1 -- start start 2024-03-28T15:06:21.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.254+0000 7eff42081640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff240039e0 0x7eff3c1f0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:21.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.254+0000 7eff42081640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff3c14ee00 0x7eff3c1516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:21.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.254+0000 7eff42081640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff3c14d7b0 0x7eff3c14f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:21.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.254+0000 7eff42081640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c0c61d0 con 0x7eff3c14d7b0 2024-03-28T15:06:21.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.254+0000 7eff42081640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c0c5ef0 con 0x7eff3c14ee00 2024-03-28T15:06:21.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.254+0000 7eff42081640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7eff3c0cc390 con 0x7eff240039e0 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.254+0000 7eff3bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff3c14d7b0 0x7eff3c14f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.254+0000 7eff3bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff3c14d7b0 0x7eff3c14f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:51848/0 (socket says 172.21.15.186:51848) 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.254+0000 7eff3bfff640 1 -- 172.21.15.186:0/2278526597 learned_addr learned my addr 172.21.15.186:0/2278526597 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.254+0000 7eff3affd640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff3c14ee00 0x7eff3c1516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff3b7fe640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff240039e0 0x7eff3c1f0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff3bfff640 1 -- 172.21.15.186:0/2278526597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff240039e0 msgr2=0x7eff3c1f0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff3bfff640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff240039e0 0x7eff3c1f0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff3bfff640 1 -- 172.21.15.186:0/2278526597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff3c14ee00 msgr2=0x7eff3c1516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff3bfff640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff3c14ee00 0x7eff3c1516a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff3bfff640 1 -- 172.21.15.186:0/2278526597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3c15ebe0 con 0x7eff3c14d7b0 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff3bfff640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff3c14d7b0 0x7eff3c14f480 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0d0ec0 tx=0x7eff3400fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:21.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff23fff640 1 -- 172.21.15.186:0/2278526597 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3401a070 con 0x7eff3c14d7b0 2024-03-28T15:06:21.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff23fff640 1 -- 172.21.15.186:0/2278526597 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff34006d60 con 0x7eff3c14d7b0 2024-03-28T15:06:21.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff42081640 1 -- 172.21.15.186:0/2278526597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff3c1f1460 con 0x7eff3c14d7b0 2024-03-28T15:06:21.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff23fff640 1 -- 172.21.15.186:0/2278526597 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff34003b90 con 0x7eff3c14d7b0 2024-03-28T15:06:21.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.255+0000 7eff42081640 1 -- 172.21.15.186:0/2278526597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff3c14c8f0 con 0x7eff3c14d7b0 2024-03-28T15:06:21.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.256+0000 7eff42081640 1 -- 172.21.15.186:0/2278526597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff3c069500 con 0x7eff3c14d7b0 2024-03-28T15:06:21.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.257+0000 7eff23fff640 1 -- 172.21.15.186:0/2278526597 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7eff3401d070 con 0x7eff3c14d7b0 2024-03-28T15:06:21.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.257+0000 7eff23fff640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7eff2806c530 0x7eff2806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:21.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.257+0000 7eff23fff640 1 -- 172.21.15.186:0/2278526597 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7eff3408acb0 con 0x7eff3c14d7b0 2024-03-28T15:06:21.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.257+0000 7eff3b7fe640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7eff2806c530 0x7eff2806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:21.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.257+0000 7eff3b7fe640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7eff2806c530 0x7eff2806e9f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7eff2c036000 tx=0x7eff2c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:21.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.262+0000 7eff23fff640 1 -- 172.21.15.186:0/2278526597 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7eff34058b00 con 0x7eff3c14d7b0 2024-03-28T15:06:21.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.475+0000 7eff42081640 1 -- 172.21.15.186:0/2278526597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff3c0cbb60 con 0x7eff3c14d7b0 2024-03-28T15:06:21.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.475+0000 7eff23fff640 1 -- 172.21.15.186:0/2278526597 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6013 (secure 0 0 0) 0x7eff3405ce00 con 0x7eff3c14d7b0 2024-03-28T15:06:21.476 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:21.476 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:18.356264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:06:21.476 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 22 2024-03-28T15:06:21.478 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 -- 172.21.15.186:0/2278526597 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7eff2806c530 msgr2=0x7eff2806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:21.478 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7eff2806c530 0x7eff2806e9f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7eff2c036000 tx=0x7eff2c030000 comp rx=0 tx=0).stop 2024-03-28T15:06:21.478 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 -- 172.21.15.186:0/2278526597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff3c14d7b0 msgr2=0x7eff3c14f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:21.478 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff3c14d7b0 0x7eff3c14f480 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0d0ec0 tx=0x7eff3400fd40 comp rx=0 tx=0).stop 2024-03-28T15:06:21.478 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 -- 172.21.15.186:0/2278526597 shutdown_connections 2024-03-28T15:06:21.478 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7eff2806c530 0x7eff2806e9f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:21.478 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff3c14d7b0 0x7eff3c14f480 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:21.478 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff3c14ee00 0x7eff3c1516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:21.478 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 --2- 172.21.15.186:0/2278526597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff240039e0 0x7eff3c1f0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:21.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 -- 172.21.15.186:0/2278526597 >> 172.21.15.186:0/2278526597 conn(0x7eff3c0731f0 msgr2=0x7eff3c1522b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:21.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 -- 172.21.15.186:0/2278526597 shutdown_connections 2024-03-28T15:06:21.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:21.477+0000 7eff42081640 1 -- 172.21.15.186:0/2278526597 wait complete. 2024-03-28T15:06:21.488 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:06:23.489 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:23.602 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.602+0000 7f89bf59e640 1 Processor -- start 2024-03-28T15:06:23.602 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.602+0000 7f89bf59e640 1 -- start start 2024-03-28T15:06:23.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89bf59e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f89c00b2f70 0x7f89c015b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:23.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89bf59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89c00ccbd0 con 0x7f89c0154920 2024-03-28T15:06:23.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89bf59e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89c00cceb0 con 0x7f89c00726d0 2024-03-28T15:06:23.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89bf59e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f89c00cb740 con 0x7f89c00b2f70 2024-03-28T15:06:23.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89be59c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f89c0154920 0x7f89c00d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:36520/0 (socket says 172.21.15.186:36520) 2024-03-28T15:06:23.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89be59c640 1 -- 172.21.15.186:0/982957860 learned_addr learned my addr 172.21.15.186:0/982957860 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:06:23.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2943010240 0 0) 0x7f89c00ccbd0 con 0x7f89c0154920 2024-03-28T15:06:23.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89bdd9b640 1 --2- 172.21.15.186:0/982957860 >> v2:172.21.15.191:3300/0 conn(0x7f89c00b2f70 0x7f89c015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:23.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89a8003520 con 0x7f89c0154920 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2438000063 0 0) 0x7f89a8003520 con 0x7f89c0154920 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 >> v2:172.21.15.191:3300/0 conn(0x7f89c00b2f70 msgr2=0x7f89c015b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89bd59a640 1 --2- 172.21.15.186:0/982957860 >> v2:172.21.15.191:3300/0 conn(0x7f89c00b2f70 0x7f89c015b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.603+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 >> v1:172.21.15.191:6789/0 conn(0x7f89c00726d0 legacy=0x7f89c00cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c00cd190 con 0x7f89c0154920 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f89b0001670 con 0x7f89c0154920 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 >> v1:172.21.15.186:6789/0 conn(0x7f89c0154920 legacy=0x7f89c00d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bd59a640 1 --2- 172.21.15.186:0/982957860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a80039e0 0x7f89a8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bd59a640 1 --2- 172.21.15.186:0/982957860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89a8004800 0x7f89a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bd59a640 1 --2- 172.21.15.186:0/982957860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89a80051c0 0x7f89a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f89c00cb740 con 0x7f89a80039e0 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f89a800c010 con 0x7f89a8004800 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f89a800c210 con 0x7f89a80051c0 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89be59c640 1 --2- 172.21.15.186:0/982957860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a80039e0 0x7f89a8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:23.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bdd9b640 1 --2- 172.21.15.186:0/982957860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89a8004800 0x7f89a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:23.607 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bed9d640 1 --2- 172.21.15.186:0/982957860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89a80051c0 0x7f89a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:23.607 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89be59c640 1 -- 172.21.15.186:0/982957860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89a80051c0 msgr2=0x7f89a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:23.607 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89be59c640 1 --2- 172.21.15.186:0/982957860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89a80051c0 0x7f89a800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:23.607 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89be59c640 1 -- 172.21.15.186:0/982957860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89a8004800 msgr2=0x7f89a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:23.607 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89be59c640 1 --2- 172.21.15.186:0/982957860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89a8004800 0x7f89a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:23.607 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89bed9d640 1 -- 172.21.15.186:0/982957860 reap_dead start 2024-03-28T15:06:23.607 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.604+0000 7f89be59c640 1 -- 172.21.15.186:0/982957860 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a8003520 con 0x7f89a80039e0 2024-03-28T15:06:23.607 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.605+0000 7f89be59c640 1 --2- 172.21.15.186:0/982957860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a80039e0 0x7f89a8003dc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f89b000c240 tx=0x7f89a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283243ced73af1b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:23.607 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.605+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0007d10 con 0x7f89a80039e0 2024-03-28T15:06:23.610 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.605+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b0030070 con 0x7f89a80039e0 2024-03-28T15:06:23.610 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.605+0000 7f89bd59a640 1 -- 172.21.15.186:0/982957860 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0005d70 con 0x7f89a80039e0 2024-03-28T15:06:23.610 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.606+0000 7f89bf59e640 1 -- 172.21.15.186:0/982957860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a80039e0 msgr2=0x7f89a8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.606+0000 7f89bf59e640 1 --2- 172.21.15.186:0/982957860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a80039e0 0x7f89a8003dc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f89b000c240 tx=0x7f89a8005830 comp rx=0 tx=0).stop 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.606+0000 7f89bf59e640 1 -- 172.21.15.186:0/982957860 shutdown_connections 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.606+0000 7f89bf59e640 1 --2- 172.21.15.186:0/982957860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a80039e0 0x7f89a8003dc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.606+0000 7f89bf59e640 1 -- 172.21.15.186:0/982957860 >> 172.21.15.186:0/982957860 conn(0x7f89c00735c0 msgr2=0x7f89c00739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.606+0000 7f89bf59e640 1 -- 172.21.15.186:0/982957860 shutdown_connections 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.606+0000 7f89bf59e640 1 -- 172.21.15.186:0/982957860 wait complete. 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.607+0000 7f89bf59e640 1 Processor -- start 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.607+0000 7f89bf59e640 1 -- start start 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bf59e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89a80039e0 0x7f89c0153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bf59e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89c00b4e00 0x7f89c014a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bf59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89c015bdf0 0x7f89c0145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bf59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f89c0154d00 con 0x7f89c015bdf0 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bf59e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f89c00cd750 con 0x7f89a80039e0 2024-03-28T15:06:23.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bf59e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f89c015b130 con 0x7f89c00b4e00 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89c015bdf0 0x7f89c0145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89c015bdf0 0x7f89c0145f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:51872/0 (socket says 172.21.15.186:51872) 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bed9d640 1 -- 172.21.15.186:0/4243016309 learned_addr learned my addr 172.21.15.186:0/4243016309 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bdd9b640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89c00b4e00 0x7f89c014a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89be59c640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89a80039e0 0x7f89c0153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bed9d640 1 -- 172.21.15.186:0/4243016309 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89c00b4e00 msgr2=0x7f89c014a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bed9d640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89c00b4e00 0x7f89c014a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bed9d640 1 -- 172.21.15.186:0/4243016309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89a80039e0 msgr2=0x7f89c0153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bed9d640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89a80039e0 0x7f89c0153b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bed9d640 1 -- 172.21.15.186:0/4243016309 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c015da60 con 0x7f89c015bdf0 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89bed9d640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89c015bdf0 0x7f89c0145f30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f89c00b34a0 tx=0x7f89b800feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.608+0000 7f89aeffd640 1 -- 172.21.15.186:0/4243016309 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8019070 con 0x7f89c015bdf0 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.609+0000 7f89aeffd640 1 -- 172.21.15.186:0/4243016309 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b8006d60 con 0x7f89c015bdf0 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.609+0000 7f89bf59e640 1 -- 172.21.15.186:0/4243016309 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c00b52d0 con 0x7f89c015bdf0 2024-03-28T15:06:23.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.609+0000 7f89aeffd640 1 -- 172.21.15.186:0/4243016309 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8003bb0 con 0x7f89c015bdf0 2024-03-28T15:06:23.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.609+0000 7f89bf59e640 1 -- 172.21.15.186:0/4243016309 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c015dfe0 con 0x7f89c015bdf0 2024-03-28T15:06:23.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.610+0000 7f89bf59e640 1 -- 172.21.15.186:0/4243016309 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c0141550 con 0x7f89c015bdf0 2024-03-28T15:06:23.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.610+0000 7f89aeffd640 1 -- 172.21.15.186:0/4243016309 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f89b801c070 con 0x7f89c015bdf0 2024-03-28T15:06:23.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.610+0000 7f89aeffd640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f89a006c530 0x7f89a006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:23.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.610+0000 7f89be59c640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f89a006c530 0x7f89a006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:23.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.610+0000 7f89aeffd640 1 -- 172.21.15.186:0/4243016309 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f89b808a9f0 con 0x7f89c015bdf0 2024-03-28T15:06:23.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.611+0000 7f89be59c640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f89a006c530 0x7f89a006e9f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f89b0034000 tx=0x7f89b0007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:23.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.615+0000 7f89aeffd640 1 -- 172.21.15.186:0/4243016309 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f89c0141550 con 0x7f89c015bdf0 2024-03-28T15:06:23.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.826+0000 7f89bf59e640 1 -- 172.21.15.186:0/4243016309 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89c0069500 con 0x7f89c015bdf0 2024-03-28T15:06:23.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.827+0000 7f89aeffd640 1 -- 172.21.15.186:0/4243016309 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6013 (secure 0 0 0) 0x7f89b8058840 con 0x7f89c015bdf0 2024-03-28T15:06:23.827 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:23.827 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:18.356264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:06:23.828 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 22 2024-03-28T15:06:23.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 -- 172.21.15.186:0/4243016309 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f89a006c530 msgr2=0x7f89a006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:23.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f89a006c530 0x7f89a006e9f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f89b0034000 tx=0x7f89b0007000 comp rx=0 tx=0).stop 2024-03-28T15:06:23.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 -- 172.21.15.186:0/4243016309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89c015bdf0 msgr2=0x7f89c0145f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:23.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89c015bdf0 0x7f89c0145f30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f89c00b34a0 tx=0x7f89b800feb0 comp rx=0 tx=0).stop 2024-03-28T15:06:23.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 -- 172.21.15.186:0/4243016309 shutdown_connections 2024-03-28T15:06:23.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f89a006c530 0x7f89a006e9f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:23.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89c015bdf0 0x7f89c0145f30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:23.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89c00b4e00 0x7f89c014a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:23.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 --2- 172.21.15.186:0/4243016309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89a80039e0 0x7f89c0153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:23.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 -- 172.21.15.186:0/4243016309 >> 172.21.15.186:0/4243016309 conn(0x7f89c00735c0 msgr2=0x7f89c01589d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:23.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 -- 172.21.15.186:0/4243016309 shutdown_connections 2024-03-28T15:06:23.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:23.829+0000 7f89bf59e640 1 -- 172.21.15.186:0/4243016309 wait complete. 2024-03-28T15:06:23.840 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:06:25.841 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:25.953 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.952+0000 7fb59c29a640 1 Processor -- start 2024-03-28T15:06:25.953 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.952+0000 7fb59c29a640 1 -- start start 2024-03-28T15:06:25.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59c29a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb594152810 0x7fb594158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:25.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59c29a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb594071340 0x7fb59406c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:25.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59c29a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb594072df0 con 0x7fb59406ce60 2024-03-28T15:06:25.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59c29a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb594159500 con 0x7fb594152810 2024-03-28T15:06:25.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59c29a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fb5940c8660 con 0x7fb594071340 2024-03-28T15:06:25.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59a036640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb594152810 0x7fb594158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59a036640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb594152810 0x7fb594158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:51888/0 (socket says 172.21.15.186:51888) 2024-03-28T15:06:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59a036640 1 -- 172.21.15.186:0/3740834139 learned_addr learned my addr 172.21.15.186:0/3740834139 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59a036640 1 -- 172.21.15.186:0/3740834139 >> v2:172.21.15.191:3300/0 conn(0x7fb594071340 msgr2=0x7fb59406c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb599835640 1 --2- 172.21.15.186:0/3740834139 >> v2:172.21.15.191:3300/0 conn(0x7fb594071340 0x7fb59406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59a036640 1 --2- 172.21.15.186:0/3740834139 >> v2:172.21.15.191:3300/0 conn(0x7fb594071340 0x7fb59406c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59a036640 1 -- 172.21.15.186:0/3740834139 >> v1:172.21.15.191:6789/0 conn(0x7fb59406ce60 legacy=0x7fb5940c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.953+0000 7fb59a036640 1 -- 172.21.15.186:0/3740834139 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb594071d50 con 0x7fb594152810 2024-03-28T15:06:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.954+0000 7fb59a036640 1 --2- 172.21.15.186:0/3740834139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb594152810 0x7fb594158fc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb590009870 tx=0x7fb59002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8daa6b3e282616c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.954+0000 7fb599034640 1 -- 172.21.15.186:0/3740834139 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb590035020 con 0x7fb594152810 2024-03-28T15:06:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.954+0000 7fb599034640 1 -- 172.21.15.186:0/3740834139 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb590031870 con 0x7fb594152810 2024-03-28T15:06:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.954+0000 7fb599034640 1 -- 172.21.15.186:0/3740834139 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb590031c40 con 0x7fb594152810 2024-03-28T15:06:25.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.955+0000 7fb59c29a640 1 -- 172.21.15.186:0/3740834139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb594152810 msgr2=0x7fb594158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:25.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.955+0000 7fb59c29a640 1 --2- 172.21.15.186:0/3740834139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb594152810 0x7fb594158fc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb590009870 tx=0x7fb59002b910 comp rx=0 tx=0).stop 2024-03-28T15:06:25.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.955+0000 7fb59c29a640 1 -- 172.21.15.186:0/3740834139 shutdown_connections 2024-03-28T15:06:25.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.955+0000 7fb59c29a640 1 --2- 172.21.15.186:0/3740834139 >> v2:172.21.15.191:3300/0 conn(0x7fb594071340 0x7fb59406c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:25.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.955+0000 7fb59c29a640 1 --2- 172.21.15.186:0/3740834139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb594152810 0x7fb594158fc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:25.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.955+0000 7fb59c29a640 1 -- 172.21.15.186:0/3740834139 >> 172.21.15.186:0/3740834139 conn(0x7fb5940c8a20 msgr2=0x7fb5940c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:25.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.955+0000 7fb59c29a640 1 -- 172.21.15.186:0/3740834139 shutdown_connections 2024-03-28T15:06:25.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.955+0000 7fb59c29a640 1 -- 172.21.15.186:0/3740834139 wait complete. 2024-03-28T15:06:25.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.956+0000 7fb59c29a640 1 Processor -- start 2024-03-28T15:06:25.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.956+0000 7fb59c29a640 1 -- start start 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59c29a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb59406ce60 0x7fb594150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59c29a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb594071340 0x7fb5940cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59c29a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb594152810 0x7fb5940cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59c29a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb5940725f0 con 0x7fb59406ce60 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59c29a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb594072310 con 0x7fb594152810 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59c29a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb5940728d0 con 0x7fb594071340 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59a036640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb59406ce60 0x7fb594150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59a036640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb59406ce60 0x7fb594150f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:51904/0 (socket says 172.21.15.186:51904) 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59a036640 1 -- 172.21.15.186:0/263689464 learned_addr learned my addr 172.21.15.186:0/263689464 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59a837640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb594152810 0x7fb5940cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb599835640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb594071340 0x7fb5940cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:25.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59a036640 1 -- 172.21.15.186:0/263689464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb594071340 msgr2=0x7fb5940cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59a036640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb594071340 0x7fb5940cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59a036640 1 -- 172.21.15.186:0/263689464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb594152810 msgr2=0x7fb5940cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59a036640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb594152810 0x7fb5940cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59a036640 1 -- 172.21.15.186:0/263689464 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5941eedb0 con 0x7fb59406ce60 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59a837640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb594152810 0x7fb5940cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb599835640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb594071340 0x7fb5940cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59a036640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb59406ce60 0x7fb594150f60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb590009870 tx=0x7fb5900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb58affd640 1 -- 172.21.15.186:0/263689464 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5900443f0 con 0x7fb59406ce60 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb58affd640 1 -- 172.21.15.186:0/263689464 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5900445b0 con 0x7fb59406ce60 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59c29a640 1 -- 172.21.15.186:0/263689464 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5941ef050 con 0x7fb59406ce60 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb58affd640 1 -- 172.21.15.186:0/263689464 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5900449a0 con 0x7fb59406ce60 2024-03-28T15:06:25.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.957+0000 7fb59c29a640 1 -- 172.21.15.186:0/263689464 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb594155a80 con 0x7fb59406ce60 2024-03-28T15:06:25.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.959+0000 7fb59c29a640 1 -- 172.21.15.186:0/263689464 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb594069500 con 0x7fb59406ce60 2024-03-28T15:06:25.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.959+0000 7fb58affd640 1 -- 172.21.15.186:0/263689464 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb59004b060 con 0x7fb59406ce60 2024-03-28T15:06:25.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.959+0000 7fb58affd640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb57806c530 0x7fb57806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:25.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.960+0000 7fb58affd640 1 -- 172.21.15.186:0/263689464 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7fb5900ada20 con 0x7fb59406ce60 2024-03-28T15:06:25.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.960+0000 7fb599835640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb57806c530 0x7fb57806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:25.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.960+0000 7fb599835640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb57806c530 0x7fb57806e9f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb5940cae90 tx=0x7fb584008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:25.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:25.964+0000 7fb58affd640 1 -- 172.21.15.186:0/263689464 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb59003f070 con 0x7fb59406ce60 2024-03-28T15:06:26.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.174+0000 7fb59c29a640 1 -- 172.21.15.186:0/263689464 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5940c77f0 con 0x7fb59406ce60 2024-03-28T15:06:26.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.175+0000 7fb58affd640 1 -- 172.21.15.186:0/263689464 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6020 (secure 0 0 0) 0x7fb59002f320 con 0x7fb59406ce60 2024-03-28T15:06:26.176 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:26.176 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:24.381272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4790,"qdb_cluster":[4790]},"id":1}]} 2024-03-28T15:06:26.176 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 23 2024-03-28T15:06:26.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 -- 172.21.15.186:0/263689464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb57806c530 msgr2=0x7fb57806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:26.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb57806c530 0x7fb57806e9f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb5940cae90 tx=0x7fb584008040 comp rx=0 tx=0).stop 2024-03-28T15:06:26.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 -- 172.21.15.186:0/263689464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb59406ce60 msgr2=0x7fb594150f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:26.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb59406ce60 0x7fb594150f60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb590009870 tx=0x7fb5900046c0 comp rx=0 tx=0).stop 2024-03-28T15:06:26.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 -- 172.21.15.186:0/263689464 shutdown_connections 2024-03-28T15:06:26.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb57806c530 0x7fb57806e9f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:26.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb594152810 0x7fb5940cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:26.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb594071340 0x7fb5940cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:26.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 --2- 172.21.15.186:0/263689464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb59406ce60 0x7fb594150f60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:26.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 -- 172.21.15.186:0/263689464 >> 172.21.15.186:0/263689464 conn(0x7fb5940c8a20 msgr2=0x7fb5940c9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:26.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 -- 172.21.15.186:0/263689464 shutdown_connections 2024-03-28T15:06:26.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:26.177+0000 7fb59c29a640 1 -- 172.21.15.186:0/263689464 wait complete. 2024-03-28T15:06:26.189 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4879 2024-03-28T15:06:26.189 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.f 2024-03-28T15:06:26.449 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:06:26.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:06:27.189 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-03-28T15:06:27.190 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-28T15:06:27.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-03-28T15:06:27.193 INFO:tasks.ceph.mds.f:Started 2024-03-28T15:06:27.193 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:06:27.277 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2024-03-28T15:06:27.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.303+0000 7f9fba688640 1 Processor -- start 2024-03-28T15:06:27.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.303+0000 7f9fba688640 1 -- start start 2024-03-28T15:06:27.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.304+0000 7f9fba688640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9fb4154bd0 0x7f9fb415a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:27.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.304+0000 7f9fba688640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9fb40c53c0 0x7f9fb40c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:27.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.304+0000 7f9fba688640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb40c57e0 0x7f9fb40cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:27.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.304+0000 7f9fba688640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f9fb40c5ca0 con 0x7f9fb4154bd0 2024-03-28T15:06:27.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.304+0000 7f9fba688640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f9fb415b080 con 0x7f9fb40c53c0 2024-03-28T15:06:27.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.304+0000 7f9fba688640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb415b360 con 0x7f9fb40c57e0 2024-03-28T15:06:27.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.304+0000 7f9fb3fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9fb4154bd0 0x7f9fb415a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:27.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.304+0000 7f9fb3fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9fb4154bd0 0x7f9fb415a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:51906/0 (socket says 172.21.15.186:51906) 2024-03-28T15:06:27.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.304+0000 7f9fb3fff640 1 -- 172.21.15.186:0/3917927909 learned_addr learned my addr 172.21.15.186:0/3917927909 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:27.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.304+0000 7f9fb37fe640 1 --2- 172.21.15.186:0/3917927909 >> v2:172.21.15.191:3300/0 conn(0x7f9fb40c53c0 0x7f9fb40c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:27.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.304+0000 7f9fb8c25640 1 --2- 172.21.15.186:0/3917927909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb40c57e0 0x7f9fb40cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:27.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.305+0000 7f9fb3fff640 1 -- 172.21.15.186:0/3917927909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb40c57e0 msgr2=0x7f9fb40cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:27.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.305+0000 7f9fb3fff640 1 --2- 172.21.15.186:0/3917927909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb40c57e0 0x7f9fb40cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.305+0000 7f9fb3fff640 1 -- 172.21.15.186:0/3917927909 >> v2:172.21.15.191:3300/0 conn(0x7f9fb40c53c0 msgr2=0x7f9fb40c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:06:27.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.305+0000 7f9fb3fff640 1 --2- 172.21.15.186:0/3917927909 >> v2:172.21.15.191:3300/0 conn(0x7f9fb40c53c0 0x7f9fb40c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.305+0000 7f9fb3fff640 1 -- 172.21.15.186:0/3917927909 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb415b640 con 0x7f9fb4154bd0 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.305+0000 7f9fb3fff640 1 --2- 172.21.15.186:0/3917927909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fb4154bd0 0x7f9fb415a7d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9fb408d6b0 tx=0x7f9fa402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e120248305fd4fed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fb2ffd640 1 -- 172.21.15.186:0/3917927909 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4034070 con 0x7f9fb4154bd0 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fb2ffd640 1 -- 172.21.15.186:0/3917927909 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa4030860 con 0x7f9fb4154bd0 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fb2ffd640 1 -- 172.21.15.186:0/3917927909 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4030c30 con 0x7f9fb4154bd0 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fba688640 1 -- 172.21.15.186:0/3917927909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fb4154bd0 msgr2=0x7f9fb415a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fba688640 1 --2- 172.21.15.186:0/3917927909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fb4154bd0 0x7f9fb415a7d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9fb408d6b0 tx=0x7f9fa402b9a0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fba688640 1 -- 172.21.15.186:0/3917927909 shutdown_connections 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fba688640 1 --2- 172.21.15.186:0/3917927909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb40c57e0 0x7f9fb40cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fba688640 1 --2- 172.21.15.186:0/3917927909 >> v2:172.21.15.191:3300/0 conn(0x7f9fb40c53c0 0x7f9fb40c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fba688640 1 --2- 172.21.15.186:0/3917927909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fb4154bd0 0x7f9fb415a7d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fba688640 1 -- 172.21.15.186:0/3917927909 >> 172.21.15.186:0/3917927909 conn(0x7f9fb40731f0 msgr2=0x7f9fb4073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fba688640 1 -- 172.21.15.186:0/3917927909 shutdown_connections 2024-03-28T15:06:27.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.306+0000 7f9fba688640 1 -- 172.21.15.186:0/3917927909 wait complete. 2024-03-28T15:06:27.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.307+0000 7f9fba688640 1 Processor -- start 2024-03-28T15:06:27.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.308+0000 7f9fba688640 1 -- start start 2024-03-28T15:06:27.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.308+0000 7f9fba688640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40c53c0 0x7f9fb415c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:27.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.308+0000 7f9fba688640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fb40c57e0 0x7f9fb41f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:27.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.308+0000 7f9fba688640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb4154bd0 0x7f9fb41f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:27.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.308+0000 7f9fba688640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb40cc390 con 0x7f9fb40c57e0 2024-03-28T15:06:27.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.308+0000 7f9fba688640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb40c61d0 con 0x7f9fb40c53c0 2024-03-28T15:06:27.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.308+0000 7f9fba688640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb415ad10 con 0x7f9fb4154bd0 2024-03-28T15:06:27.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.308+0000 7f9fb3fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40c53c0 0x7f9fb415c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:27.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.308+0000 7f9fb37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fb40c57e0 0x7f9fb41f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:27.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.308+0000 7f9fb3fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40c53c0 0x7f9fb415c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:33796/0 (socket says 172.21.15.186:33796) 2024-03-28T15:06:27.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.308+0000 7f9fb3fff640 1 -- 172.21.15.186:0/3758186276 learned_addr learned my addr 172.21.15.186:0/3758186276 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:06:27.312 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fb8c25640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb4154bd0 0x7f9fb41f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fb3fff640 1 -- 172.21.15.186:0/3758186276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb4154bd0 msgr2=0x7f9fb41f4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fb3fff640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb4154bd0 0x7f9fb41f4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fb3fff640 1 -- 172.21.15.186:0/3758186276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fb40c57e0 msgr2=0x7f9fb41f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fb3fff640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fb40c57e0 0x7f9fb41f3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fb3fff640 1 -- 172.21.15.186:0/3758186276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb41e77e0 con 0x7f9fb40c53c0 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fb37fe640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fb40c57e0 0x7f9fb41f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fb3fff640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40c53c0 0x7f9fb415c3d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9fa402b9e0 tx=0x7f9fa40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fb0ff9640 1 -- 172.21.15.186:0/3758186276 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4034040 con 0x7f9fb40c53c0 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fb0ff9640 1 -- 172.21.15.186:0/3758186276 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa4039d90 con 0x7f9fb40c53c0 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fba688640 1 -- 172.21.15.186:0/3758186276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb4153f90 con 0x7f9fb40c53c0 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fb0ff9640 1 -- 172.21.15.186:0/3758186276 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4043610 con 0x7f9fb40c53c0 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.309+0000 7f9fba688640 1 -- 172.21.15.186:0/3758186276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb415df00 con 0x7f9fb40c53c0 2024-03-28T15:06:27.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.310+0000 7f9fba688640 1 -- 172.21.15.186:0/3758186276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fb4069500 con 0x7f9fb40c53c0 2024-03-28T15:06:27.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.311+0000 7f9fb0ff9640 1 -- 172.21.15.186:0/3758186276 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9fa4037070 con 0x7f9fb40c53c0 2024-03-28T15:06:27.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.311+0000 7f9fb0ff9640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9fa006c560 0x7f9fa006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:06:27.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.311+0000 7f9fb0ff9640 1 -- 172.21.15.186:0/3758186276 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f9fa40ad900 con 0x7f9fb40c53c0 2024-03-28T15:06:27.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.311+0000 7f9fb37fe640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9fa006c560 0x7f9fa006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:06:27.315 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.312+0000 7f9fb37fe640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9fa006c560 0x7f9fa006ea20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40686d0 tx=0x7f9fa8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:06:27.316 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.316+0000 7f9fb0ff9640 1 -- 172.21.15.186:0/3758186276 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f9fa403e070 con 0x7f9fb40c53c0 2024-03-28T15:06:27.531 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.530+0000 7f9fba688640 1 -- 172.21.15.186:0/3758186276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fb40cb550 con 0x7f9fb40c53c0 2024-03-28T15:06:27.532 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.531+0000 7f9fb0ff9640 1 -- 172.21.15.186:0/3758186276 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6848 (secure 0 0 0) 0x7f9fa407b830 con 0x7f9fb40c53c0 2024-03-28T15:06:27.532 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:06:27.532 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5137,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:24.381272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4790,"qdb_cluster":[4790]},"id":1}]} 2024-03-28T15:06:27.533 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 24 2024-03-28T15:06:27.534 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 -- 172.21.15.186:0/3758186276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9fa006c560 msgr2=0x7f9fa006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:27.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9fa006c560 0x7f9fa006ea20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40686d0 tx=0x7f9fa8008040 comp rx=0 tx=0).stop 2024-03-28T15:06:27.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 -- 172.21.15.186:0/3758186276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40c53c0 msgr2=0x7f9fb415c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:06:27.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40c53c0 0x7f9fb415c3d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9fa402b9e0 tx=0x7f9fa40046c0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 -- 172.21.15.186:0/3758186276 shutdown_connections 2024-03-28T15:06:27.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9fa006c560 0x7f9fa006ea20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb4154bd0 0x7f9fb41f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fb40c57e0 0x7f9fb41f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 --2- 172.21.15.186:0/3758186276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40c53c0 0x7f9fb415c3d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:06:27.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 -- 172.21.15.186:0/3758186276 >> 172.21.15.186:0/3758186276 conn(0x7f9fb40731f0 msgr2=0x7f9fb40cd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:06:27.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 -- 172.21.15.186:0/3758186276 shutdown_connections 2024-03-28T15:06:27.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:06:27.533+0000 7f9fba688640 1 -- 172.21.15.186:0/3758186276 wait complete. 2024-03-28T15:06:27.545 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-03-28T15:06:27.545 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 104 secs before thrashing 2024-03-28T15:06:29.555 INFO:tasks.ceph.mds.b.smithi191.stderr:2024-03-28T15:06:29.555+0000 7f18bf7ea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T15:06:30.037 INFO:tasks.workunit.client.0.smithi186.stderr:error: RPC failed; curl 56 OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0 2024-03-28T15:06:30.041 INFO:tasks.workunit.client.0.smithi186.stderr:fetch-pack: unexpected disconnect while reading sideband packet 2024-03-28T15:06:30.046 INFO:tasks.workunit.client.0.smithi186.stderr:fatal: early EOF 2024-03-28T15:06:30.046 INFO:tasks.workunit.client.0.smithi186.stderr:fatal: fetch-pack: invalid index-pack output 2024-03-28T15:06:30.187 INFO:tasks.workunit.client.0.smithi186.stderr:+ retry 2024-03-28T15:06:30.187 INFO:tasks.workunit.client.0.smithi186.stderr:+ rm -rf ceph 2024-03-28T15:06:30.188 INFO:tasks.workunit.client.0.smithi186.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git 2024-03-28T15:06:30.192 INFO:tasks.workunit.client.0.smithi186.stderr:Cloning into 'ceph'... 2024-03-28T15:06:56.563 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:06:56.566 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:07:26.641 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:07:26.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:07:56.750 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:07:56.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:08:11.546 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:11.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.668+0000 7f975fa47640 1 Processor -- start 2024-03-28T15:08:11.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.668+0000 7f975fa47640 1 -- start start 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.669+0000 7f975fa47640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9758151990 0x7f97580c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.669+0000 7f975fa47640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f97580b1e30 0x7f97580c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.669+0000 7f975fa47640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f975806caa0 con 0x7f97580c2d00 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.669+0000 7f975fa47640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f97580c1da0 con 0x7f97580b1e30 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.669+0000 7f975fa47640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f975806cd80 con 0x7f9758151990 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.670+0000 7f975dfe4640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f97580c2d00 0x7f97580712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:44688/0 (socket says 172.21.15.186:44688) 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.670+0000 7f975dfe4640 1 -- 172.21.15.186:0/4032667070 learned_addr learned my addr 172.21.15.186:0/4032667070 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.670+0000 7f975d7e3640 1 --2- 172.21.15.186:0/4032667070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9758151990 0x7f97580c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.670+0000 7f975cfe2640 1 --2- 172.21.15.186:0/4032667070 >> v2:172.21.15.191:3300/0 conn(0x7f97580b1e30 0x7f97580c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.670+0000 7f974ffff640 1 -- 172.21.15.186:0/4032667070 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1757175386 0 0) 0x7f975806caa0 con 0x7f97580c2d00 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.670+0000 7f974ffff640 1 -- 172.21.15.186:0/4032667070 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9744003520 con 0x7f97580c2d00 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.670+0000 7f975d7e3640 1 -- 172.21.15.186:0/4032667070 >> v2:172.21.15.191:3300/0 conn(0x7f97580b1e30 msgr2=0x7f97580c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:11.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.670+0000 7f975d7e3640 1 --2- 172.21.15.186:0/4032667070 >> v2:172.21.15.191:3300/0 conn(0x7f97580b1e30 0x7f97580c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:11.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.670+0000 7f975d7e3640 1 -- 172.21.15.186:0/4032667070 >> v1:172.21.15.186:6789/0 conn(0x7f97580c2d00 legacy=0x7f97580712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:11.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.670+0000 7f975d7e3640 1 -- 172.21.15.186:0/4032667070 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97580c1340 con 0x7f9758151990 2024-03-28T15:08:11.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.670+0000 7f975cfe2640 1 --2- 172.21.15.186:0/4032667070 >> v2:172.21.15.191:3300/0 conn(0x7f97580b1e30 0x7f97580c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:08:11.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.671+0000 7f975d7e3640 1 --2- 172.21.15.186:0/4032667070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9758151990 0x7f97580c2230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f975400b730 tx=0x7f975402bbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=470ac4dc0bdacb10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:11.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.671+0000 7f974ffff640 1 -- 172.21.15.186:0/4032667070 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754039070 con 0x7f9758151990 2024-03-28T15:08:11.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.671+0000 7f974ffff640 1 -- 172.21.15.186:0/4032667070 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975402c9a0 con 0x7f9758151990 2024-03-28T15:08:11.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.671+0000 7f974ffff640 1 -- 172.21.15.186:0/4032667070 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975402cd50 con 0x7f9758151990 2024-03-28T15:08:11.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.672+0000 7f975fa47640 1 -- 172.21.15.186:0/4032667070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9758151990 msgr2=0x7f97580c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:11.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.672+0000 7f975fa47640 1 --2- 172.21.15.186:0/4032667070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9758151990 0x7f97580c2230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f975400b730 tx=0x7f975402bbf0 comp rx=0 tx=0).stop 2024-03-28T15:08:11.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.672+0000 7f975fa47640 1 -- 172.21.15.186:0/4032667070 shutdown_connections 2024-03-28T15:08:11.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.672+0000 7f975fa47640 1 --2- 172.21.15.186:0/4032667070 >> v2:172.21.15.191:3300/0 conn(0x7f97580b1e30 0x7f97580c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:11.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.672+0000 7f975fa47640 1 --2- 172.21.15.186:0/4032667070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9758151990 0x7f97580c2230 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:11.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.672+0000 7f975fa47640 1 -- 172.21.15.186:0/4032667070 >> 172.21.15.186:0/4032667070 conn(0x7f97580c8a20 msgr2=0x7f97580c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:11.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.672+0000 7f975fa47640 1 -- 172.21.15.186:0/4032667070 shutdown_connections 2024-03-28T15:08:11.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.672+0000 7f975fa47640 1 -- 172.21.15.186:0/4032667070 wait complete. 2024-03-28T15:08:11.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.673+0000 7f975fa47640 1 Processor -- start 2024-03-28T15:08:11.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.673+0000 7f975fa47640 1 -- start start 2024-03-28T15:08:11.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975fa47640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97580b1e30 0x7f97580cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:11.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975fa47640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97580c2d00 0x7f97580cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:11.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975fa47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9758151990 0x7f97581e0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:11.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975fa47640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9758073dd0 con 0x7f9758151990 2024-03-28T15:08:11.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975fa47640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f97580c1900 con 0x7f97580c2d00 2024-03-28T15:08:11.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975fa47640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f9758073f50 con 0x7f97580b1e30 2024-03-28T15:08:11.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975dfe4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9758151990 0x7f97581e0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:11.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975dfe4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9758151990 0x7f97581e0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37298/0 (socket says 172.21.15.186:37298) 2024-03-28T15:08:11.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975dfe4640 1 -- 172.21.15.186:0/904073585 learned_addr learned my addr 172.21.15.186:0/904073585 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975cfe2640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97580c2d00 0x7f97580cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975d7e3640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97580b1e30 0x7f97580cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975dfe4640 1 -- 172.21.15.186:0/904073585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97580b1e30 msgr2=0x7f97580cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975dfe4640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97580b1e30 0x7f97580cafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975dfe4640 1 -- 172.21.15.186:0/904073585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97580c2d00 msgr2=0x7f97580cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975dfe4640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97580c2d00 0x7f97580cb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.674+0000 7f975dfe4640 1 -- 172.21.15.186:0/904073585 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97581ec8c0 con 0x7f9758151990 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.675+0000 7f975dfe4640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9758151990 0x7f97581e0c00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f975000caa0 tx=0x7f975000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.675+0000 7f974dffb640 1 -- 172.21.15.186:0/904073585 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9750017070 con 0x7f9758151990 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.675+0000 7f974dffb640 1 -- 172.21.15.186:0/904073585 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975000ac00 con 0x7f9758151990 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.675+0000 7f974dffb640 1 -- 172.21.15.186:0/904073585 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9750012400 con 0x7f9758151990 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.675+0000 7f975fa47640 1 -- 172.21.15.186:0/904073585 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97581ecb80 con 0x7f9758151990 2024-03-28T15:08:11.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.675+0000 7f975fa47640 1 -- 172.21.15.186:0/904073585 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9758158a80 con 0x7f9758151990 2024-03-28T15:08:11.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.676+0000 7f975fa47640 1 -- 172.21.15.186:0/904073585 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9758069500 con 0x7f9758151990 2024-03-28T15:08:11.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.677+0000 7f974dffb640 1 -- 172.21.15.186:0/904073585 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f975001a070 con 0x7f9758151990 2024-03-28T15:08:11.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.677+0000 7f974dffb640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f973c06c530 0x7f973c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:11.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.677+0000 7f974dffb640 1 -- 172.21.15.186:0/904073585 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f975008b570 con 0x7f9758151990 2024-03-28T15:08:11.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.677+0000 7f975d7e3640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f973c06c530 0x7f973c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:11.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.678+0000 7f975d7e3640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f973c06c530 0x7f973c06e9f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9754036000 tx=0x7f9754030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:11.683 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.683+0000 7f974dffb640 1 -- 172.21.15.186:0/904073585 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f97500593c0 con 0x7f9758151990 2024-03-28T15:08:11.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.922+0000 7f975fa47640 1 -- 172.21.15.186:0/904073585 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97580b1a00 con 0x7f9758151990 2024-03-28T15:08:11.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.923+0000 7f974dffb640 1 -- 172.21.15.186:0/904073585 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6848 (secure 0 0 0) 0x7f975005d6c0 con 0x7f9758151990 2024-03-28T15:08:11.923 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:11.923 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5137,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:24.381272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4790,"qdb_cluster":[4790]},"id":1}]} 2024-03-28T15:08:11.924 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 24 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 -- 172.21.15.186:0/904073585 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f973c06c530 msgr2=0x7f973c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f973c06c530 0x7f973c06e9f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9754036000 tx=0x7f9754030000 comp rx=0 tx=0).stop 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 -- 172.21.15.186:0/904073585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9758151990 msgr2=0x7f97581e0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9758151990 0x7f97581e0c00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f975000caa0 tx=0x7f975000cf60 comp rx=0 tx=0).stop 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 -- 172.21.15.186:0/904073585 shutdown_connections 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f973c06c530 0x7f973c06e9f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9758151990 0x7f97581e0c00 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97580c2d00 0x7f97580cb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 --2- 172.21.15.186:0/904073585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97580b1e30 0x7f97580cafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 -- 172.21.15.186:0/904073585 >> 172.21.15.186:0/904073585 conn(0x7f97580c8a20 msgr2=0x7f975814f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 -- 172.21.15.186:0/904073585 shutdown_connections 2024-03-28T15:08:11.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:11.925+0000 7f975fa47640 1 -- 172.21.15.186:0/904073585 wait complete. 2024-03-28T15:08:11.938 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2024-03-28T15:08:11.938 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T15:08:11.938 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:08:11.949 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T15:08:11.949 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:12.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.059+0000 7fa2b5593640 1 Processor -- start 2024-03-28T15:08:12.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.059+0000 7fa2b5593640 1 -- start start 2024-03-28T15:08:12.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2b5593640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa2b006c370 0x7fa2b00c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:12.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2b5593640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2b006b7d0 con 0x7fa2b006c750 2024-03-28T15:08:12.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2b5593640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2b006bab0 con 0x7fa2b00c9510 2024-03-28T15:08:12.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2b5593640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fa2b015c5c0 con 0x7fa2b006c370 2024-03-28T15:08:12.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2af7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fa2b006c750 0x7fa2b0163240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:44702/0 (socket says 172.21.15.186:44702) 2024-03-28T15:08:12.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2af7fe640 1 -- 172.21.15.186:0/3856935162 learned_addr learned my addr 172.21.15.186:0/3856935162 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2aeffd640 1 --2- 172.21.15.186:0/3856935162 >> v2:172.21.15.191:3300/0 conn(0x7fa2b006c370 0x7fa2b00c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2242717089 0 0) 0x7fa2b006b7d0 con 0x7fa2b006c750 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa298003520 con 0x7fa2b006c750 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4246857382 0 0) 0x7fa2b006bab0 con 0x7fa2b00c9510 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2b006b7d0 con 0x7fa2b00c9510 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2073066626 0 0) 0x7fa298003520 con 0x7fa2b006c750 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 >> v2:172.21.15.191:3300/0 conn(0x7fa2b006c370 msgr2=0x7fa2b00c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 --2- 172.21.15.186:0/3856935162 >> v2:172.21.15.191:3300/0 conn(0x7fa2b006c370 0x7fa2b00c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 >> v1:172.21.15.191:6789/0 conn(0x7fa2b00c9510 legacy=0x7fa2b015beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b0158c10 con 0x7fa2b006c750 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7fa2a80019f0 con 0x7fa2b006c750 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 >> v1:172.21.15.186:6789/0 conn(0x7fa2b006c750 legacy=0x7fa2b0163240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:12.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa298003b40 0x7fa298003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298004800 0x7fa298004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2980051c0 0x7fa29800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b015c5c0 con 0x7fa298004800 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa29800bfb0 con 0x7fa298003b40 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa29800c1b0 con 0x7fa2980051c0 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2aeffd640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa298003b40 0x7fa298003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.060+0000 7fa2ae7fc640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298004800 0x7fa298004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.061+0000 7fa2af7fe640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2980051c0 0x7fa29800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.061+0000 7fa2aeffd640 1 -- 172.21.15.186:0/3856935162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2980051c0 msgr2=0x7fa29800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.061+0000 7fa2aeffd640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2980051c0 0x7fa29800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.061+0000 7fa2aeffd640 1 -- 172.21.15.186:0/3856935162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298004800 msgr2=0x7fa298004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.061+0000 7fa2aeffd640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298004800 0x7fa298004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:12.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.061+0000 7fa2aeffd640 1 -- 172.21.15.186:0/3856935162 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa298003520 con 0x7fa298003b40 2024-03-28T15:08:12.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.061+0000 7fa2af7fe640 1 -- 172.21.15.186:0/3856935162 reap_dead start 2024-03-28T15:08:12.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.061+0000 7fa2ae7fc640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298004800 0x7fa298004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:08:12.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.062+0000 7fa2aeffd640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa298003b40 0x7fa298003f40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009ac0 tx=0x7fa298005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9404bf893766a79 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:12.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.062+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0039070 con 0x7fa298003b40 2024-03-28T15:08:12.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.062+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2a002c660 con 0x7fa298003b40 2024-03-28T15:08:12.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.062+0000 7fa2adffb640 1 -- 172.21.15.186:0/3856935162 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0009690 con 0x7fa298003b40 2024-03-28T15:08:12.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.063+0000 7fa2b5593640 1 -- 172.21.15.186:0/3856935162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa298003b40 msgr2=0x7fa298003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:12.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.063+0000 7fa2b5593640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa298003b40 0x7fa298003f40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009ac0 tx=0x7fa298005830 comp rx=0 tx=0).stop 2024-03-28T15:08:12.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.063+0000 7fa2b5593640 1 -- 172.21.15.186:0/3856935162 shutdown_connections 2024-03-28T15:08:12.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.063+0000 7fa2b5593640 1 --2- 172.21.15.186:0/3856935162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa298003b40 0x7fa298003f40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:12.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.063+0000 7fa2b5593640 1 -- 172.21.15.186:0/3856935162 >> 172.21.15.186:0/3856935162 conn(0x7fa2b0070020 msgr2=0x7fa2b0070400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:12.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.063+0000 7fa2b5593640 1 -- 172.21.15.186:0/3856935162 shutdown_connections 2024-03-28T15:08:12.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.063+0000 7fa2b5593640 1 -- 172.21.15.186:0/3856935162 wait complete. 2024-03-28T15:08:12.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.064+0000 7fa2b5593640 1 Processor -- start 2024-03-28T15:08:12.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.064+0000 7fa2b5593640 1 -- start start 2024-03-28T15:08:12.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2b5593640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298003b40 0x7fa2b0150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:12.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2b5593640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2b0150050 0x7fa2b014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:12.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2b5593640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2b014fb00 0x7fa2b014f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:12.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2b5593640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b006cbc0 con 0x7fa298003b40 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2b5593640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b0159260 con 0x7fa2b014fb00 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2b5593640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b01588c0 con 0x7fa2b0150050 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2aeffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298003b40 0x7fa2b0150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2aeffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298003b40 0x7fa2b0150920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37308/0 (socket says 172.21.15.186:37308) 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2aeffd640 1 -- 172.21.15.186:0/2603414738 learned_addr learned my addr 172.21.15.186:0/2603414738 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2af7fe640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2b014fb00 0x7fa2b014f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2ae7fc640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2b0150050 0x7fa2b014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2aeffd640 1 -- 172.21.15.186:0/2603414738 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2b0150050 msgr2=0x7fa2b014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2aeffd640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2b0150050 0x7fa2b014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2aeffd640 1 -- 172.21.15.186:0/2603414738 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2b014fb00 msgr2=0x7fa2b014f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2aeffd640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2b014fb00 0x7fa2b014f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2aeffd640 1 -- 172.21.15.186:0/2603414738 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b01e3960 con 0x7fa298003b40 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.065+0000 7fa2aeffd640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298003b40 0x7fa2b0150920 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009ac0 tx=0x7fa2a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:12.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.066+0000 7fa2977fe640 1 -- 172.21.15.186:0/2603414738 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0039050 con 0x7fa298003b40 2024-03-28T15:08:12.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.066+0000 7fa2977fe640 1 -- 172.21.15.186:0/2603414738 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2a0045070 con 0x7fa298003b40 2024-03-28T15:08:12.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.066+0000 7fa2b5593640 1 -- 172.21.15.186:0/2603414738 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b00c4770 con 0x7fa298003b40 2024-03-28T15:08:12.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.066+0000 7fa2977fe640 1 -- 172.21.15.186:0/2603414738 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a00044e0 con 0x7fa298003b40 2024-03-28T15:08:12.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.066+0000 7fa2b5593640 1 -- 172.21.15.186:0/2603414738 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b00c4d30 con 0x7fa298003b40 2024-03-28T15:08:12.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.067+0000 7fa2b5593640 1 -- 172.21.15.186:0/2603414738 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b00686d0 con 0x7fa298003b40 2024-03-28T15:08:12.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.067+0000 7fa2977fe640 1 -- 172.21.15.186:0/2603414738 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa2a0003830 con 0x7fa298003b40 2024-03-28T15:08:12.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.068+0000 7fa2977fe640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa29c06c560 0x7fa29c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:12.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.068+0000 7fa2ae7fc640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa29c06c560 0x7fa29c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:12.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.068+0000 7fa2977fe640 1 -- 172.21.15.186:0/2603414738 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7fa2a00ad640 con 0x7fa298003b40 2024-03-28T15:08:12.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.068+0000 7fa2ae7fc640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa29c06c560 0x7fa29c06ea20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0151440 tx=0x7fa2a4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:12.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.072+0000 7fa2977fe640 1 -- 172.21.15.186:0/2603414738 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa2a004f020 con 0x7fa298003b40 2024-03-28T15:08:12.354 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.354+0000 7fa2b5593640 1 -- 172.21.15.186:0/2603414738 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2b006c150 con 0x7fa298003b40 2024-03-28T15:08:12.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.354+0000 7fa2977fe640 1 -- 172.21.15.186:0/2603414738 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6848 (secure 0 0 0) 0x7fa2a007bbb0 con 0x7fa298003b40 2024-03-28T15:08:12.355 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:12.355 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5137,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:24.381272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4790,"qdb_cluster":[4790]},"id":1}]} 2024-03-28T15:08:12.355 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 24 2024-03-28T15:08:12.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.356+0000 7fa2b5593640 1 -- 172.21.15.186:0/2603414738 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa29c06c560 msgr2=0x7fa29c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:12.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.356+0000 7fa2b5593640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa29c06c560 0x7fa29c06ea20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0151440 tx=0x7fa2a4009040 comp rx=0 tx=0).stop 2024-03-28T15:08:12.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.356+0000 7fa2b5593640 1 -- 172.21.15.186:0/2603414738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298003b40 msgr2=0x7fa2b0150920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:12.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.356+0000 7fa2b5593640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298003b40 0x7fa2b0150920 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009ac0 tx=0x7fa2a0004730 comp rx=0 tx=0).stop 2024-03-28T15:08:12.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.357+0000 7fa2b5593640 1 -- 172.21.15.186:0/2603414738 shutdown_connections 2024-03-28T15:08:12.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.357+0000 7fa2b5593640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa29c06c560 0x7fa29c06ea20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:12.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.357+0000 7fa2b5593640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2b014fb00 0x7fa2b014f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:12.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.357+0000 7fa2b5593640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2b0150050 0x7fa2b014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:12.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.357+0000 7fa2b5593640 1 --2- 172.21.15.186:0/2603414738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa298003b40 0x7fa2b0150920 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:12.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.357+0000 7fa2b5593640 1 -- 172.21.15.186:0/2603414738 >> 172.21.15.186:0/2603414738 conn(0x7fa2b0070020 msgr2=0x7fa2b00734e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:12.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.357+0000 7fa2b5593640 1 -- 172.21.15.186:0/2603414738 shutdown_connections 2024-03-28T15:08:12.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:12.357+0000 7fa2b5593640 1 -- 172.21.15.186:0/2603414738 wait complete. 2024-03-28T15:08:12.368 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:08:14.369 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:14.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.489+0000 7f70b1e22640 1 Processor -- start 2024-03-28T15:08:14.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.489+0000 7f70b1e22640 1 -- start start 2024-03-28T15:08:14.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.490+0000 7f70b1e22640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 0x7f70ac0c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:14.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.490+0000 7f70b1e22640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f70ac157490 0x7f70ac0c8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:14.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.490+0000 7f70b1e22640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70ac0c5430 con 0x7f70ac072470 2024-03-28T15:08:14.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.490+0000 7f70b1e22640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f70ac151c40 con 0x7f70ac157490 2024-03-28T15:08:14.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.490+0000 7f70b1e22640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f70ac0c56d0 con 0x7f70ac151860 2024-03-28T15:08:14.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.491+0000 7f70ab7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 0x7f70ac0c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:14.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.491+0000 7f70abfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f70ac157490 0x7f70ac0c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:14.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.491+0000 7f70ab7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 0x7f70ac0c3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:40806/0 (socket says 172.21.15.186:40806) 2024-03-28T15:08:14.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.491+0000 7f70ab7fe640 1 -- 172.21.15.186:0/1241924799 learned_addr learned my addr 172.21.15.186:0/1241924799 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:14.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.491+0000 7f70abfff640 1 -- 172.21.15.186:0/1241924799 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 msgr2=0x7f70ac0c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:14.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.491+0000 7f70abfff640 1 --2- 172.21.15.186:0/1241924799 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 0x7f70ac0c3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:14.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.491+0000 7f70abfff640 1 -- 172.21.15.186:0/1241924799 >> v1:172.21.15.191:6789/0 conn(0x7f70ac072470 legacy=0x7f70ac0c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:14.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.491+0000 7f70abfff640 1 -- 172.21.15.186:0/1241924799 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ac0c59b0 con 0x7f70ac157490 2024-03-28T15:08:14.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.491+0000 7f70aa7fc640 1 -- 172.21.15.186:0/1241924799 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1906314679 0 0) 0x7f70ac0c5430 con 0x7f70ac072470 2024-03-28T15:08:14.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.491+0000 7f70ab7fe640 1 --2- 172.21.15.186:0/1241924799 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 0x7f70ac0c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:08:14.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.492+0000 7f70abfff640 1 --2- 172.21.15.186:0/1241924799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70ac157490 0x7f70ac0c8100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f70a4009d90 tx=0x7f70a402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51179f8a87795750 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:14.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.492+0000 7f70aa7fc640 1 -- 172.21.15.186:0/1241924799 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4039070 con 0x7f70ac157490 2024-03-28T15:08:14.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.492+0000 7f70aa7fc640 1 -- 172.21.15.186:0/1241924799 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a402cce0 con 0x7f70ac157490 2024-03-28T15:08:14.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.492+0000 7f70aa7fc640 1 -- 172.21.15.186:0/1241924799 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4034660 con 0x7f70ac157490 2024-03-28T15:08:14.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.493+0000 7f70b1e22640 1 -- 172.21.15.186:0/1241924799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70ac157490 msgr2=0x7f70ac0c8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:14.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.493+0000 7f70b1e22640 1 --2- 172.21.15.186:0/1241924799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70ac157490 0x7f70ac0c8100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f70a4009d90 tx=0x7f70a402bf30 comp rx=0 tx=0).stop 2024-03-28T15:08:14.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.493+0000 7f70b1e22640 1 -- 172.21.15.186:0/1241924799 shutdown_connections 2024-03-28T15:08:14.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.493+0000 7f70b1e22640 1 --2- 172.21.15.186:0/1241924799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70ac157490 0x7f70ac0c8100 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:14.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.493+0000 7f70b1e22640 1 --2- 172.21.15.186:0/1241924799 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 0x7f70ac0c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:14.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.493+0000 7f70b1e22640 1 -- 172.21.15.186:0/1241924799 >> 172.21.15.186:0/1241924799 conn(0x7f70ac06f750 msgr2=0x7f70ac06fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:14.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.493+0000 7f70b1e22640 1 -- 172.21.15.186:0/1241924799 shutdown_connections 2024-03-28T15:08:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.493+0000 7f70b1e22640 1 -- 172.21.15.186:0/1241924799 wait complete. 2024-03-28T15:08:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.494+0000 7f70b1e22640 1 Processor -- start 2024-03-28T15:08:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.494+0000 7f70b1e22640 1 -- start start 2024-03-28T15:08:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.495+0000 7f70b1e22640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac072470 0x7f70ac14cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.495+0000 7f70b1e22640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 0x7f70ac1f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.495+0000 7f70b1e22640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70ac157490 0x7f70ac0c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.495+0000 7f70b1e22640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac0c9640 con 0x7f70ac072470 2024-03-28T15:08:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.495+0000 7f70b1e22640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac071f50 con 0x7f70ac157490 2024-03-28T15:08:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.495+0000 7f70b1e22640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f70ac157900 con 0x7f70ac151860 2024-03-28T15:08:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.495+0000 7f70ab7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac072470 0x7f70ac14cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.495+0000 7f70aaffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 0x7f70ac1f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.495+0000 7f70ab7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac072470 0x7f70ac14cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37324/0 (socket says 172.21.15.186:37324) 2024-03-28T15:08:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.495+0000 7f70ab7fe640 1 -- 172.21.15.186:0/3486613464 learned_addr learned my addr 172.21.15.186:0/3486613464 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:14.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.495+0000 7f70abfff640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70ac157490 0x7f70ac0c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:14.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f70ab7fe640 1 -- 172.21.15.186:0/3486613464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 msgr2=0x7f70ac1f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:14.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f70ab7fe640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 0x7f70ac1f3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:14.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f70ab7fe640 1 -- 172.21.15.186:0/3486613464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70ac157490 msgr2=0x7f70ac0c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:14.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f70ab7fe640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70ac157490 0x7f70ac0c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:14.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f70ab7fe640 1 -- 172.21.15.186:0/3486613464 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ac1e2690 con 0x7f70ac072470 2024-03-28T15:08:14.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f70aaffd640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 0x7f70ac1f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:08:14.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f70ab7fe640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac072470 0x7f70ac14cbc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f709c00ea00 tx=0x7f709c00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:14.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f7093fff640 1 -- 172.21.15.186:0/3486613464 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c017070 con 0x7f70ac072470 2024-03-28T15:08:14.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f7093fff640 1 -- 172.21.15.186:0/3486613464 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f709c00ca60 con 0x7f70ac072470 2024-03-28T15:08:14.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f70b1e22640 1 -- 172.21.15.186:0/3486613464 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70ac1e2970 con 0x7f70ac072470 2024-03-28T15:08:14.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f7093fff640 1 -- 172.21.15.186:0/3486613464 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c00ce10 con 0x7f70ac072470 2024-03-28T15:08:14.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.496+0000 7f70b1e22640 1 -- 172.21.15.186:0/3486613464 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70ac1f0680 con 0x7f70ac072470 2024-03-28T15:08:14.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.497+0000 7f70b1e22640 1 -- 172.21.15.186:0/3486613464 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70ac0686d0 con 0x7f70ac072470 2024-03-28T15:08:14.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.497+0000 7f7093fff640 1 -- 172.21.15.186:0/3486613464 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f709c01a070 con 0x7f70ac072470 2024-03-28T15:08:14.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.498+0000 7f7093fff640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f709806c560 0x7f709806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:14.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.498+0000 7f7093fff640 1 -- 172.21.15.186:0/3486613464 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f709c08baf0 con 0x7f70ac072470 2024-03-28T15:08:14.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.498+0000 7f70aaffd640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f709806c560 0x7f709806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:14.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.498+0000 7f70aaffd640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f709806c560 0x7f709806ea20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f70ac1e1b60 tx=0x7f70a0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:14.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.503+0000 7f7093fff640 1 -- 172.21.15.186:0/3486613464 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f709c0598b0 con 0x7f70ac072470 2024-03-28T15:08:14.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.791+0000 7f70b1e22640 1 -- 172.21.15.186:0/3486613464 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70ac0c3ba0 con 0x7f70ac072470 2024-03-28T15:08:14.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.792+0000 7f7093fff640 1 -- 172.21.15.186:0/3486613464 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6848 (secure 0 0 0) 0x7f709c05dbb0 con 0x7f70ac072470 2024-03-28T15:08:14.792 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:14.793 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5137,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:24.381272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4790,"qdb_cluster":[4790]},"id":1}]} 2024-03-28T15:08:14.793 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 24 2024-03-28T15:08:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.794+0000 7f70b1e22640 1 -- 172.21.15.186:0/3486613464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f709806c560 msgr2=0x7f709806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.794+0000 7f70b1e22640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f709806c560 0x7f709806ea20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f70ac1e1b60 tx=0x7f70a0009040 comp rx=0 tx=0).stop 2024-03-28T15:08:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.794+0000 7f70b1e22640 1 -- 172.21.15.186:0/3486613464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac072470 msgr2=0x7f70ac14cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.794+0000 7f70b1e22640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac072470 0x7f70ac14cbc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f709c00ea00 tx=0x7f709c00eec0 comp rx=0 tx=0).stop 2024-03-28T15:08:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.794+0000 7f70b1e22640 1 -- 172.21.15.186:0/3486613464 shutdown_connections 2024-03-28T15:08:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.794+0000 7f70b1e22640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f709806c560 0x7f709806ea20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.794+0000 7f70b1e22640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70ac157490 0x7f70ac0c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.794+0000 7f70b1e22640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac151860 0x7f70ac1f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.794+0000 7f70b1e22640 1 --2- 172.21.15.186:0/3486613464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac072470 0x7f70ac14cbc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.794+0000 7f70b1e22640 1 -- 172.21.15.186:0/3486613464 >> 172.21.15.186:0/3486613464 conn(0x7f70ac06f750 msgr2=0x7f70ac14dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.795+0000 7f70b1e22640 1 -- 172.21.15.186:0/3486613464 shutdown_connections 2024-03-28T15:08:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:14.795+0000 7f70b1e22640 1 -- 172.21.15.186:0/3486613464 wait complete. 2024-03-28T15:08:14.808 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:08:16.809 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:16.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.939+0000 7f75a86b8640 1 Processor -- start 2024-03-28T15:08:16.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.939+0000 7f75a86b8640 1 -- start start 2024-03-28T15:08:16.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a86b8640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f75a00ce9e0 0x7f75a006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:16.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a86b8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a00705b0 0x7f75a015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:16.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a86b8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75a00c85b0 con 0x7f75a01528b0 2024-03-28T15:08:16.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a86b8640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f75a00c52f0 con 0x7f75a00ce9e0 2024-03-28T15:08:16.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a86b8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f75a00c4d70 con 0x7f75a00705b0 2024-03-28T15:08:16.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a6c55640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a00705b0 0x7f75a015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:16.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a6c55640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a00705b0 0x7f75a015bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:40836/0 (socket says 172.21.15.186:40836) 2024-03-28T15:08:16.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a5c53640 1 --2- 172.21.15.186:0/552241721 >> v2:172.21.15.191:3300/0 conn(0x7f75a00ce9e0 0x7f75a006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:16.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a6c55640 1 -- 172.21.15.186:0/552241721 learned_addr learned my addr 172.21.15.186:0/552241721 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:16.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a6c55640 1 -- 172.21.15.186:0/552241721 >> v2:172.21.15.191:3300/0 conn(0x7f75a00ce9e0 msgr2=0x7f75a006fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:16.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a6c55640 1 --2- 172.21.15.186:0/552241721 >> v2:172.21.15.191:3300/0 conn(0x7f75a00ce9e0 0x7f75a006fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:16.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a6c55640 1 -- 172.21.15.186:0/552241721 >> v1:172.21.15.191:6789/0 conn(0x7f75a01528b0 legacy=0x7f75a00c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:16.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a6c55640 1 -- 172.21.15.186:0/552241721 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a00c3810 con 0x7f75a00705b0 2024-03-28T15:08:16.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.941+0000 7f75a5c53640 1 --2- 172.21.15.186:0/552241721 >> v2:172.21.15.191:3300/0 conn(0x7f75a00ce9e0 0x7f75a006fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:08:16.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.942+0000 7f75a6c55640 1 --2- 172.21.15.186:0/552241721 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a00705b0 0x7f75a015bfe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7598009e50 tx=0x7f759802bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c79b99bfd2aacd17 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:16.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.942+0000 7f75a5452640 1 -- 172.21.15.186:0/552241721 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759802ca80 con 0x7f75a00705b0 2024-03-28T15:08:16.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.942+0000 7f75a5452640 1 -- 172.21.15.186:0/552241721 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f759802cc20 con 0x7f75a00705b0 2024-03-28T15:08:16.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.942+0000 7f75a5452640 1 -- 172.21.15.186:0/552241721 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75980343f0 con 0x7f75a00705b0 2024-03-28T15:08:16.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.943+0000 7f75a86b8640 1 -- 172.21.15.186:0/552241721 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a00705b0 msgr2=0x7f75a015bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:16.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.943+0000 7f75a86b8640 1 --2- 172.21.15.186:0/552241721 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a00705b0 0x7f75a015bfe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7598009e50 tx=0x7f759802bee0 comp rx=0 tx=0).stop 2024-03-28T15:08:16.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.943+0000 7f75a86b8640 1 -- 172.21.15.186:0/552241721 shutdown_connections 2024-03-28T15:08:16.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.943+0000 7f75a86b8640 1 --2- 172.21.15.186:0/552241721 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a00705b0 0x7f75a015bfe0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:16.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.943+0000 7f75a86b8640 1 --2- 172.21.15.186:0/552241721 >> v2:172.21.15.191:3300/0 conn(0x7f75a00ce9e0 0x7f75a006fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:16.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.943+0000 7f75a86b8640 1 -- 172.21.15.186:0/552241721 >> 172.21.15.186:0/552241721 conn(0x7f75a006d670 msgr2=0x7f75a006da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:16.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.943+0000 7f75a86b8640 1 -- 172.21.15.186:0/552241721 shutdown_connections 2024-03-28T15:08:16.945 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.943+0000 7f75a86b8640 1 -- 172.21.15.186:0/552241721 wait complete. 2024-03-28T15:08:16.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.945+0000 7f75a86b8640 1 Processor -- start 2024-03-28T15:08:16.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.945+0000 7f75a86b8640 1 -- start start 2024-03-28T15:08:16.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a86b8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75a00705b0 0x7f75a00ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:16.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a86b8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75a00ce9e0 0x7f75a01f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:16.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a86b8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a01528b0 0x7f75a0071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a86b8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75a00c3530 con 0x7f75a00ce9e0 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a86b8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f75a00c3250 con 0x7f75a00705b0 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a86b8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f75a0152c90 con 0x7f75a01528b0 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a5c53640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75a00ce9e0 0x7f75a01f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a5c53640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75a00ce9e0 0x7f75a01f4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37340/0 (socket says 172.21.15.186:37340) 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a5c53640 1 -- 172.21.15.186:0/2761678545 learned_addr learned my addr 172.21.15.186:0/2761678545 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a6454640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75a00705b0 0x7f75a00ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a6c55640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a01528b0 0x7f75a0071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a5c53640 1 -- 172.21.15.186:0/2761678545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a01528b0 msgr2=0x7f75a0071370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a5c53640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a01528b0 0x7f75a0071370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a5c53640 1 -- 172.21.15.186:0/2761678545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75a00705b0 msgr2=0x7f75a00ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a5c53640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75a00705b0 0x7f75a00ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a5c53640 1 -- 172.21.15.186:0/2761678545 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a00cd7a0 con 0x7f75a00ce9e0 2024-03-28T15:08:16.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a6454640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75a00705b0 0x7f75a00ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:08:16.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.946+0000 7f75a6c55640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a01528b0 0x7f75a0071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:08:16.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.947+0000 7f75a5c53640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75a00ce9e0 0x7f75a01f4ca0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f759000e9e0 tx=0x7f759000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:16.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.947+0000 7f7596ffd640 1 -- 172.21.15.186:0/2761678545 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7590005990 con 0x7f75a00ce9e0 2024-03-28T15:08:16.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.947+0000 7f7596ffd640 1 -- 172.21.15.186:0/2761678545 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7590005b30 con 0x7f75a00ce9e0 2024-03-28T15:08:16.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.947+0000 7f75a86b8640 1 -- 172.21.15.186:0/2761678545 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a00c89d0 con 0x7f75a00ce9e0 2024-03-28T15:08:16.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.947+0000 7f7596ffd640 1 -- 172.21.15.186:0/2761678545 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759000c9d0 con 0x7f75a00ce9e0 2024-03-28T15:08:16.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.947+0000 7f75a86b8640 1 -- 172.21.15.186:0/2761678545 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a00cd4c0 con 0x7f75a00ce9e0 2024-03-28T15:08:16.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.949+0000 7f75a86b8640 1 -- 172.21.15.186:0/2761678545 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a0069500 con 0x7f75a00ce9e0 2024-03-28T15:08:16.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.949+0000 7f7596ffd640 1 -- 172.21.15.186:0/2761678545 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f759000cb70 con 0x7f75a00ce9e0 2024-03-28T15:08:16.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.949+0000 7f7596ffd640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f758406c560 0x7f758406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:16.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.950+0000 7f7596ffd640 1 -- 172.21.15.186:0/2761678545 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f759008bc40 con 0x7f75a00ce9e0 2024-03-28T15:08:16.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.950+0000 7f75a6454640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f758406c560 0x7f758406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:16.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.950+0000 7f75a6454640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f758406c560 0x7f758406ea20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f759c007fd0 tx=0x7f759c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:16.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:16.957+0000 7f7596ffd640 1 -- 172.21.15.186:0/2761678545 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7590017020 con 0x7f75a00ce9e0 2024-03-28T15:08:17.255 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.255+0000 7f75a86b8640 1 -- 172.21.15.186:0/2761678545 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75a0070390 con 0x7f75a00ce9e0 2024-03-28T15:08:17.256 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.255+0000 7f7596ffd640 1 -- 172.21.15.186:0/2761678545 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6848 (secure 0 0 0) 0x7f759000ce60 con 0x7f75a00ce9e0 2024-03-28T15:08:17.256 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:17.256 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5137,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:24.381272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4790,"qdb_cluster":[4790]},"id":1}]} 2024-03-28T15:08:17.256 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 24 2024-03-28T15:08:17.258 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 -- 172.21.15.186:0/2761678545 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f758406c560 msgr2=0x7f758406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:17.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f758406c560 0x7f758406ea20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f759c007fd0 tx=0x7f759c009040 comp rx=0 tx=0).stop 2024-03-28T15:08:17.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 -- 172.21.15.186:0/2761678545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75a00ce9e0 msgr2=0x7f75a01f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:17.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75a00ce9e0 0x7f75a01f4ca0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f759000e9e0 tx=0x7f759000eea0 comp rx=0 tx=0).stop 2024-03-28T15:08:17.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 -- 172.21.15.186:0/2761678545 shutdown_connections 2024-03-28T15:08:17.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f758406c560 0x7f758406ea20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:17.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a01528b0 0x7f75a0071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:17.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75a00ce9e0 0x7f75a01f4ca0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:17.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 --2- 172.21.15.186:0/2761678545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75a00705b0 0x7f75a00ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:17.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 -- 172.21.15.186:0/2761678545 >> 172.21.15.186:0/2761678545 conn(0x7f75a006d670 msgr2=0x7f75a0147b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:17.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 -- 172.21.15.186:0/2761678545 shutdown_connections 2024-03-28T15:08:17.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:17.258+0000 7f75a86b8640 1 -- 172.21.15.186:0/2761678545 wait complete. 2024-03-28T15:08:17.270 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:08:19.272 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:19.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.401+0000 7f48e93d0640 1 Processor -- start 2024-03-28T15:08:19.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.402+0000 7f48e93d0640 1 -- start start 2024-03-28T15:08:19.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e93d0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f48e4154bd0 0x7f48e415a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:19.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e93d0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48e40c6e00 con 0x7f48e40c57e0 2024-03-28T15:08:19.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e93d0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48e415b080 con 0x7f48e40c53c0 2024-03-28T15:08:19.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e93d0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f48e40c5ca0 con 0x7f48e4154bd0 2024-03-28T15:08:19.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e37fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f48e40c57e0 0x7f48e40cb040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51652/0 (socket says 172.21.15.186:51652) 2024-03-28T15:08:19.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e37fe640 1 -- 172.21.15.186:0/2909571016 learned_addr learned my addr 172.21.15.186:0/2909571016 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:08:19.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e2ffd640 1 --2- 172.21.15.186:0/2909571016 >> v2:172.21.15.191:3300/0 conn(0x7f48e4154bd0 0x7f48e415a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:19.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e1ffb640 1 -- 172.21.15.186:0/2909571016 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3521067797 0 0) 0x7f48e40c6e00 con 0x7f48e40c57e0 2024-03-28T15:08:19.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e1ffb640 1 -- 172.21.15.186:0/2909571016 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48cc003520 con 0x7f48e40c57e0 2024-03-28T15:08:19.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e2ffd640 1 -- 172.21.15.186:0/2909571016 >> v1:172.21.15.191:6789/0 conn(0x7f48e40c53c0 legacy=0x7f48e40c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:19.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e1ffb640 1 -- 172.21.15.186:0/2909571016 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2033005153 0 0) 0x7f48cc003520 con 0x7f48e40c57e0 2024-03-28T15:08:19.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.403+0000 7f48e2ffd640 1 -- 172.21.15.186:0/2909571016 >> v1:172.21.15.186:6789/0 conn(0x7f48e40c57e0 legacy=0x7f48e40cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:19.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.404+0000 7f48e2ffd640 1 -- 172.21.15.186:0/2909571016 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e415b360 con 0x7f48e4154bd0 2024-03-28T15:08:19.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.405+0000 7f48e2ffd640 1 --2- 172.21.15.186:0/2909571016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48e4154bd0 0x7f48e415a7d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f48d40097a0 tx=0x7f48d402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfa4b0136bb85607 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:19.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.405+0000 7f48e1ffb640 1 -- 172.21.15.186:0/2909571016 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4037070 con 0x7f48e4154bd0 2024-03-28T15:08:19.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.405+0000 7f48e1ffb640 1 -- 172.21.15.186:0/2909571016 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d4030510 con 0x7f48e4154bd0 2024-03-28T15:08:19.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.405+0000 7f48e1ffb640 1 -- 172.21.15.186:0/2909571016 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc0019f0 con 0x7f48e4154bd0 2024-03-28T15:08:19.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.406+0000 7f48e93d0640 1 -- 172.21.15.186:0/2909571016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48e4154bd0 msgr2=0x7f48e415a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:19.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.406+0000 7f48e93d0640 1 --2- 172.21.15.186:0/2909571016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48e4154bd0 0x7f48e415a7d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f48d40097a0 tx=0x7f48d402b710 comp rx=0 tx=0).stop 2024-03-28T15:08:19.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.406+0000 7f48e93d0640 1 -- 172.21.15.186:0/2909571016 shutdown_connections 2024-03-28T15:08:19.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.406+0000 7f48e93d0640 1 --2- 172.21.15.186:0/2909571016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48e4154bd0 0x7f48e415a7d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:19.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.406+0000 7f48e93d0640 1 -- 172.21.15.186:0/2909571016 >> 172.21.15.186:0/2909571016 conn(0x7f48e40731f0 msgr2=0x7f48e4073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:19.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.406+0000 7f48e93d0640 1 -- 172.21.15.186:0/2909571016 shutdown_connections 2024-03-28T15:08:19.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.406+0000 7f48e93d0640 1 -- 172.21.15.186:0/2909571016 wait complete. 2024-03-28T15:08:19.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.407+0000 7f48e93d0640 1 Processor -- start 2024-03-28T15:08:19.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.407+0000 7f48e93d0640 1 -- start start 2024-03-28T15:08:19.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.408+0000 7f48e93d0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48e40c53c0 0x7f48e4153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:19.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.408+0000 7f48e93d0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48e40c57e0 0x7f48e415bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:19.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.408+0000 7f48e93d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48e4154bd0 0x7f48e40b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:19.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.408+0000 7f48e93d0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48e40c61d0 con 0x7f48e40c57e0 2024-03-28T15:08:19.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.408+0000 7f48e93d0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f48e40c5ef0 con 0x7f48e40c53c0 2024-03-28T15:08:19.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.408+0000 7f48e93d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f48e40cc390 con 0x7f48e4154bd0 2024-03-28T15:08:19.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.408+0000 7f48e27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48e40c57e0 0x7f48e415bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:19.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.408+0000 7f48e27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48e40c57e0 0x7f48e415bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44902/0 (socket says 172.21.15.186:44902) 2024-03-28T15:08:19.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.408+0000 7f48e27fc640 1 -- 172.21.15.186:0/3564673276 learned_addr learned my addr 172.21.15.186:0/3564673276 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:19.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.408+0000 7f48e2ffd640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48e40c53c0 0x7f48e4153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:19.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.408+0000 7f48e37fe640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48e4154bd0 0x7f48e40b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48e2ffd640 1 -- 172.21.15.186:0/3564673276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48e4154bd0 msgr2=0x7f48e40b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48e2ffd640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48e4154bd0 0x7f48e40b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48e2ffd640 1 -- 172.21.15.186:0/3564673276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48e40c57e0 msgr2=0x7f48e415bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48e2ffd640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48e40c57e0 0x7f48e415bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48e2ffd640 1 -- 172.21.15.186:0/3564673276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e415c530 con 0x7f48e40c53c0 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48e27fc640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48e40c57e0 0x7f48e415bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48e2ffd640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48e40c53c0 0x7f48e4153490 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f48e408d7d0 tx=0x7f48d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48cb7fe640 1 -- 172.21.15.186:0/3564673276 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4037040 con 0x7f48e40c53c0 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48cb7fe640 1 -- 172.21.15.186:0/3564673276 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d4004230 con 0x7f48e40c53c0 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48cb7fe640 1 -- 172.21.15.186:0/3564673276 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d403e670 con 0x7f48e40c53c0 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48e93d0640 1 -- 172.21.15.186:0/3564673276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48e415c7d0 con 0x7f48e40c53c0 2024-03-28T15:08:19.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.409+0000 7f48e93d0640 1 -- 172.21.15.186:0/3564673276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48e41531b0 con 0x7f48e40c53c0 2024-03-28T15:08:19.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.410+0000 7f48e93d0640 1 -- 172.21.15.186:0/3564673276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48e4069500 con 0x7f48e40c53c0 2024-03-28T15:08:19.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.411+0000 7f48cb7fe640 1 -- 172.21.15.186:0/3564673276 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f48d403a040 con 0x7f48e40c53c0 2024-03-28T15:08:19.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.411+0000 7f48cb7fe640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48d006c530 0x7f48d006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:19.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.411+0000 7f48e27fc640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48d006c530 0x7f48d006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:19.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.412+0000 7f48cb7fe640 1 -- 172.21.15.186:0/3564673276 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f48d40ad2d0 con 0x7f48e40c53c0 2024-03-28T15:08:19.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.412+0000 7f48e27fc640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48d006c530 0x7f48d006e9f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f48e41f46a0 tx=0x7f48d8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:19.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.416+0000 7f48cb7fe640 1 -- 172.21.15.186:0/3564673276 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f48d4042020 con 0x7f48e40c53c0 2024-03-28T15:08:19.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.736+0000 7f48e93d0640 1 -- 172.21.15.186:0/3564673276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48e40cbb60 con 0x7f48e40c53c0 2024-03-28T15:08:19.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.737+0000 7f48cb7fe640 1 -- 172.21.15.186:0/3564673276 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6848 (secure 0 0 0) 0x7f48d407aff0 con 0x7f48e40c53c0 2024-03-28T15:08:19.739 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:19.739 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5137,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:24.381272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4790,"qdb_cluster":[4790]},"id":1}]} 2024-03-28T15:08:19.739 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 24 2024-03-28T15:08:19.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 -- 172.21.15.186:0/3564673276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48d006c530 msgr2=0x7f48d006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:19.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48d006c530 0x7f48d006e9f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f48e41f46a0 tx=0x7f48d8009040 comp rx=0 tx=0).stop 2024-03-28T15:08:19.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 -- 172.21.15.186:0/3564673276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48e40c53c0 msgr2=0x7f48e4153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:19.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48e40c53c0 0x7f48e4153490 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f48e408d7d0 tx=0x7f48d40046c0 comp rx=0 tx=0).stop 2024-03-28T15:08:19.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 -- 172.21.15.186:0/3564673276 shutdown_connections 2024-03-28T15:08:19.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48d006c530 0x7f48d006e9f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:19.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48e4154bd0 0x7f48e40b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:19.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48e40c57e0 0x7f48e415bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:19.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 --2- 172.21.15.186:0/3564673276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48e40c53c0 0x7f48e4153490 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:19.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 -- 172.21.15.186:0/3564673276 >> 172.21.15.186:0/3564673276 conn(0x7f48e40731f0 msgr2=0x7f48e40b5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:19.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 -- 172.21.15.186:0/3564673276 shutdown_connections 2024-03-28T15:08:19.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:19.739+0000 7f48e93d0640 1 -- 172.21.15.186:0/3564673276 wait complete. 2024-03-28T15:08:19.756 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:08:21.757 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:21.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.881+0000 7f5274815640 1 Processor -- start 2024-03-28T15:08:21.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.881+0000 7f5274815640 1 -- start start 2024-03-28T15:08:21.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.882+0000 7f5274815640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f527006d670 0x7f527006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:21.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.882+0000 7f5274815640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006da50 0x7f527006dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:21.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.882+0000 7f5274815640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f527006f770 con 0x7f527006fb90 2024-03-28T15:08:21.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.882+0000 7f5274815640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f52700c91a0 con 0x7f527006d670 2024-03-28T15:08:21.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.882+0000 7f5274815640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f52700c4c60 con 0x7f527006da50 2024-03-28T15:08:21.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.882+0000 7f526e59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f527006d670 0x7f527006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:21.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.882+0000 7f526e59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f527006d670 0x7f527006f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44908/0 (socket says 172.21.15.186:44908) 2024-03-28T15:08:21.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.882+0000 7f526e59c640 1 -- 172.21.15.186:0/3978506379 learned_addr learned my addr 172.21.15.186:0/3978506379 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:21.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.882+0000 7f526ed9d640 1 --2- 172.21.15.186:0/3978506379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006da50 0x7f527006dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:21.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.882+0000 7f526dd9b640 1 --1- 172.21.15.186:0/3978506379 >> v1:172.21.15.191:6789/0 conn(0x7f527006fb90 0x7f52700c8a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:58638/0 (socket says 172.21.15.186:58638) 2024-03-28T15:08:21.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.883+0000 7f526ed9d640 1 -- 172.21.15.186:0/3978506379 >> v2:172.21.15.186:3300/0 conn(0x7f527006d670 msgr2=0x7f527006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:21.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.883+0000 7f526d59a640 1 -- 172.21.15.186:0/3978506379 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2337702482 0 0) 0x7f527006f770 con 0x7f527006fb90 2024-03-28T15:08:21.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.883+0000 7f526ed9d640 1 --2- 172.21.15.186:0/3978506379 >> v2:172.21.15.186:3300/0 conn(0x7f527006d670 0x7f527006f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:21.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.883+0000 7f526ed9d640 1 -- 172.21.15.186:0/3978506379 >> v1:172.21.15.191:6789/0 conn(0x7f527006fb90 legacy=0x7f52700c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:21.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.883+0000 7f526ed9d640 1 -- 172.21.15.186:0/3978506379 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5270159500 con 0x7f527006da50 2024-03-28T15:08:21.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.884+0000 7f526ed9d640 1 --2- 172.21.15.186:0/3978506379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006da50 0x7f527006dee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f526800f370 tx=0x7f5268030210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61f251d8cfef1ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:21.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.884+0000 7f526d59a640 1 -- 172.21.15.186:0/3978506379 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526800d0d0 con 0x7f527006da50 2024-03-28T15:08:21.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.884+0000 7f526d59a640 1 -- 172.21.15.186:0/3978506379 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f526800d270 con 0x7f527006da50 2024-03-28T15:08:21.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.884+0000 7f526d59a640 1 -- 172.21.15.186:0/3978506379 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526800b740 con 0x7f527006da50 2024-03-28T15:08:21.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.885+0000 7f5274815640 1 -- 172.21.15.186:0/3978506379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006da50 msgr2=0x7f527006dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:21.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.885+0000 7f5274815640 1 --2- 172.21.15.186:0/3978506379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006da50 0x7f527006dee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f526800f370 tx=0x7f5268030210 comp rx=0 tx=0).stop 2024-03-28T15:08:21.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.885+0000 7f5274815640 1 -- 172.21.15.186:0/3978506379 shutdown_connections 2024-03-28T15:08:21.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.885+0000 7f5274815640 1 --2- 172.21.15.186:0/3978506379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006da50 0x7f527006dee0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:21.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.885+0000 7f5274815640 1 --2- 172.21.15.186:0/3978506379 >> v2:172.21.15.186:3300/0 conn(0x7f527006d670 0x7f527006f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:21.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.885+0000 7f5274815640 1 -- 172.21.15.186:0/3978506379 >> 172.21.15.186:0/3978506379 conn(0x7f5270067670 msgr2=0x7f5270071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:21.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.885+0000 7f5274815640 1 -- 172.21.15.186:0/3978506379 shutdown_connections 2024-03-28T15:08:21.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.885+0000 7f5274815640 1 -- 172.21.15.186:0/3978506379 wait complete. 2024-03-28T15:08:21.887 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.887+0000 7f5274815640 1 Processor -- start 2024-03-28T15:08:21.887 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.887+0000 7f5274815640 1 -- start start 2024-03-28T15:08:21.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f5274815640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006d670 0x7f5270144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:21.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f5274815640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f527006da50 0x7f52701ef920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:21.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f5274815640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527006fb90 0x7f52701e2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:21.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f5274815640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52700c34c0 con 0x7f527006da50 2024-03-28T15:08:21.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f5274815640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f52700c31e0 con 0x7f527006fb90 2024-03-28T15:08:21.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f5274815640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f52700c50f0 con 0x7f527006d670 2024-03-28T15:08:21.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f526dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f527006da50 0x7f52701ef920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:21.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f526dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f527006da50 0x7f52701ef920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44924/0 (socket says 172.21.15.186:44924) 2024-03-28T15:08:21.890 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f526dd9b640 1 -- 172.21.15.186:0/2726455607 learned_addr learned my addr 172.21.15.186:0/2726455607 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:21.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f526e59c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006d670 0x7f5270144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:21.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f526ed9d640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527006fb90 0x7f52701e2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:21.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f526e59c640 1 -- 172.21.15.186:0/2726455607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527006fb90 msgr2=0x7f52701e2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:21.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f526e59c640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527006fb90 0x7f52701e2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:21.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f526e59c640 1 -- 172.21.15.186:0/2726455607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f527006da50 msgr2=0x7f52701ef920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:21.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f526e59c640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f527006da50 0x7f52701ef920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:21.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f526e59c640 1 -- 172.21.15.186:0/2726455607 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f527006d3f0 con 0x7f527006d670 2024-03-28T15:08:21.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.888+0000 7f526e59c640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006d670 0x7f5270144630 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f526000e8a0 tx=0x7f526000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:21.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.889+0000 7f525effd640 1 -- 172.21.15.186:0/2726455607 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5260017070 con 0x7f527006d670 2024-03-28T15:08:21.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.889+0000 7f525effd640 1 -- 172.21.15.186:0/2726455607 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f526000c940 con 0x7f527006d670 2024-03-28T15:08:21.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.889+0000 7f525effd640 1 -- 172.21.15.186:0/2726455607 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5260009e60 con 0x7f527006d670 2024-03-28T15:08:21.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.889+0000 7f5274815640 1 -- 172.21.15.186:0/2726455607 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5270157f00 con 0x7f527006d670 2024-03-28T15:08:21.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.889+0000 7f5274815640 1 -- 172.21.15.186:0/2726455607 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f52701584c0 con 0x7f527006d670 2024-03-28T15:08:21.893 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.890+0000 7f5274815640 1 -- 172.21.15.186:0/2726455607 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52700686d0 con 0x7f527006d670 2024-03-28T15:08:21.893 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.891+0000 7f525effd640 1 -- 172.21.15.186:0/2726455607 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f526001a070 con 0x7f527006d670 2024-03-28T15:08:21.893 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.891+0000 7f525effd640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f525006c560 0x7f525006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:21.893 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.892+0000 7f525effd640 1 -- 172.21.15.186:0/2726455607 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f526008b4a0 con 0x7f527006d670 2024-03-28T15:08:21.893 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.892+0000 7f526dd9b640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f525006c560 0x7f525006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:21.893 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.892+0000 7f526dd9b640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f525006c560 0x7f525006ea20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f52701e3850 tx=0x7f5264007ea0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:21.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:21.896+0000 7f525effd640 1 -- 172.21.15.186:0/2726455607 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5260059380 con 0x7f527006d670 2024-03-28T15:08:22.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.241+0000 7f5274815640 1 -- 172.21.15.186:0/2726455607 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f527006f970 con 0x7f527006d670 2024-03-28T15:08:22.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.242+0000 7f525effd640 1 -- 172.21.15.186:0/2726455607 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6848 (secure 0 0 0) 0x7f526005d680 con 0x7f527006d670 2024-03-28T15:08:22.243 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:22.243 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5137,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:24.381272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4790,"qdb_cluster":[4790]},"id":1}]} 2024-03-28T15:08:22.243 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 24 2024-03-28T15:08:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.244+0000 7f5274815640 1 -- 172.21.15.186:0/2726455607 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f525006c560 msgr2=0x7f525006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.244+0000 7f5274815640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f525006c560 0x7f525006ea20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f52701e3850 tx=0x7f5264007ea0 comp rx=0 tx=0).stop 2024-03-28T15:08:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.244+0000 7f5274815640 1 -- 172.21.15.186:0/2726455607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006d670 msgr2=0x7f5270144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.244+0000 7f5274815640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006d670 0x7f5270144630 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f526000e8a0 tx=0x7f526000ed60 comp rx=0 tx=0).stop 2024-03-28T15:08:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.245+0000 7f5274815640 1 -- 172.21.15.186:0/2726455607 shutdown_connections 2024-03-28T15:08:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.245+0000 7f5274815640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f525006c560 0x7f525006ea20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.245+0000 7f5274815640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527006fb90 0x7f52701e2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.245+0000 7f5274815640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f527006da50 0x7f52701ef920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.245+0000 7f5274815640 1 --2- 172.21.15.186:0/2726455607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f527006d670 0x7f5270144630 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.245+0000 7f5274815640 1 -- 172.21.15.186:0/2726455607 >> 172.21.15.186:0/2726455607 conn(0x7f5270067670 msgr2=0x7f52700cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.245+0000 7f5274815640 1 -- 172.21.15.186:0/2726455607 shutdown_connections 2024-03-28T15:08:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:22.245+0000 7f5274815640 1 -- 172.21.15.186:0/2726455607 wait complete. 2024-03-28T15:08:22.259 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:08:24.263 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:24.373 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.372+0000 7fdac03b5640 1 Processor -- start 2024-03-28T15:08:24.373 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.372+0000 7fdac03b5640 1 -- start start 2024-03-28T15:08:24.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.373+0000 7fdac03b5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdab8154970 0x7fdab815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:24.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.373+0000 7fdac03b5640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdab80caae0 0x7fdab80d0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:24.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.373+0000 7fdac03b5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdab80cbac0 con 0x7fdab80d0990 2024-03-28T15:08:24.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.373+0000 7fdac03b5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdab80cb870 con 0x7fdab8154970 2024-03-28T15:08:24.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.373+0000 7fdac03b5640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fdab80cc6d0 con 0x7fdab80caae0 2024-03-28T15:08:24.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.374+0000 7fdabe151640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdab8154970 0x7fdab815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:24.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.374+0000 7fdabe151640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdab8154970 0x7fdab815b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44938/0 (socket says 172.21.15.186:44938) 2024-03-28T15:08:24.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.374+0000 7fdabe151640 1 -- 172.21.15.186:0/3210263474 learned_addr learned my addr 172.21.15.186:0/3210263474 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:24.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.374+0000 7fdabd950640 1 --2- 172.21.15.186:0/3210263474 >> v2:172.21.15.191:3300/0 conn(0x7fdab80caae0 0x7fdab80d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:24.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.374+0000 7fdabe151640 1 -- 172.21.15.186:0/3210263474 >> v2:172.21.15.191:3300/0 conn(0x7fdab80caae0 msgr2=0x7fdab80d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:24.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.374+0000 7fdabe151640 1 --2- 172.21.15.186:0/3210263474 >> v2:172.21.15.191:3300/0 conn(0x7fdab80caae0 0x7fdab80d0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:24.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.374+0000 7fdabe151640 1 -- 172.21.15.186:0/3210263474 >> v1:172.21.15.191:6789/0 conn(0x7fdab80d0990 legacy=0x7fdab80ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:24.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.374+0000 7fdabe151640 1 -- 172.21.15.186:0/3210263474 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab80cbda0 con 0x7fdab8154970 2024-03-28T15:08:24.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.375+0000 7fdabe151640 1 --2- 172.21.15.186:0/3210263474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdab8154970 0x7fdab815b1d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdab80b2c80 tx=0x7fdab402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd73aa0773adc0c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:24.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.375+0000 7fdabd14f640 1 -- 172.21.15.186:0/3210263474 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4034070 con 0x7fdab8154970 2024-03-28T15:08:24.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.375+0000 7fdabd14f640 1 -- 172.21.15.186:0/3210263474 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab4030860 con 0x7fdab8154970 2024-03-28T15:08:24.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.375+0000 7fdabd14f640 1 -- 172.21.15.186:0/3210263474 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4030c30 con 0x7fdab8154970 2024-03-28T15:08:24.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.376+0000 7fdac03b5640 1 -- 172.21.15.186:0/3210263474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdab8154970 msgr2=0x7fdab815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:24.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.376+0000 7fdac03b5640 1 --2- 172.21.15.186:0/3210263474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdab8154970 0x7fdab815b1d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdab80b2c80 tx=0x7fdab402b9a0 comp rx=0 tx=0).stop 2024-03-28T15:08:24.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.376+0000 7fdac03b5640 1 -- 172.21.15.186:0/3210263474 shutdown_connections 2024-03-28T15:08:24.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.376+0000 7fdac03b5640 1 --2- 172.21.15.186:0/3210263474 >> v2:172.21.15.191:3300/0 conn(0x7fdab80caae0 0x7fdab80d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:24.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.376+0000 7fdac03b5640 1 --2- 172.21.15.186:0/3210263474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdab8154970 0x7fdab815b1d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:24.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.376+0000 7fdac03b5640 1 -- 172.21.15.186:0/3210263474 >> 172.21.15.186:0/3210263474 conn(0x7fdab8073400 msgr2=0x7fdab8073810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:24.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.376+0000 7fdac03b5640 1 -- 172.21.15.186:0/3210263474 shutdown_connections 2024-03-28T15:08:24.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.376+0000 7fdac03b5640 1 -- 172.21.15.186:0/3210263474 wait complete. 2024-03-28T15:08:24.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.377+0000 7fdac03b5640 1 Processor -- start 2024-03-28T15:08:24.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.377+0000 7fdac03b5640 1 -- start start 2024-03-28T15:08:24.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdac03b5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdab80caae0 0x7fdab8156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:24.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdac03b5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdab80d0990 0x7fdab80c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:24.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdac03b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdab8154970 0x7fdab814dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:24.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdabe151640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdab80caae0 0x7fdab8156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:24.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdac03b5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdab80729d0 con 0x7fdab80caae0 2024-03-28T15:08:24.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdac03b5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdab80cc360 con 0x7fdab80d0990 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdac03b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fdab80caf00 con 0x7fdab8154970 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdabe151640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdab80caae0 0x7fdab8156400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:44952/0 (socket says 172.21.15.186:44952) 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdabe151640 1 -- 172.21.15.186:0/2820675582 learned_addr learned my addr 172.21.15.186:0/2820675582 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdabd950640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdab80d0990 0x7fdab80c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdabe952640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdab8154970 0x7fdab814dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdabe151640 1 -- 172.21.15.186:0/2820675582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdab8154970 msgr2=0x7fdab814dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdabe151640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdab8154970 0x7fdab814dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdabe151640 1 -- 172.21.15.186:0/2820675582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdab80d0990 msgr2=0x7fdab80c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdabe151640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdab80d0990 0x7fdab80c9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdabe151640 1 -- 172.21.15.186:0/2820675582 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab80b4b20 con 0x7fdab80caae0 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.378+0000 7fdabe151640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdab80caae0 0x7fdab8156400 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdab402b9e0 tx=0x7fdab4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.379+0000 7fdaaeffd640 1 -- 172.21.15.186:0/2820675582 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4048070 con 0x7fdab80caae0 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.379+0000 7fdaaeffd640 1 -- 172.21.15.186:0/2820675582 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab4030660 con 0x7fdab80caae0 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.379+0000 7fdaaeffd640 1 -- 172.21.15.186:0/2820675582 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4034040 con 0x7fdab80caae0 2024-03-28T15:08:24.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.379+0000 7fdac03b5640 1 -- 172.21.15.186:0/2820675582 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab81f37d0 con 0x7fdab80caae0 2024-03-28T15:08:24.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.379+0000 7fdac03b5640 1 -- 172.21.15.186:0/2820675582 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab81f3d90 con 0x7fdab80caae0 2024-03-28T15:08:24.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.380+0000 7fdac03b5640 1 -- 172.21.15.186:0/2820675582 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab8069500 con 0x7fdab80caae0 2024-03-28T15:08:24.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.380+0000 7fdaaeffd640 1 -- 172.21.15.186:0/2820675582 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdab4037070 con 0x7fdab80caae0 2024-03-28T15:08:24.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.381+0000 7fdaaeffd640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fda9c06c530 0x7fda9c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:24.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.381+0000 7fdaaeffd640 1 -- 172.21.15.186:0/2820675582 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7fdab40ad580 con 0x7fdab80caae0 2024-03-28T15:08:24.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.381+0000 7fdabd950640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fda9c06c530 0x7fda9c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:24.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.381+0000 7fdabd950640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fda9c06c530 0x7fda9c06e9f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdab8151b80 tx=0x7fdaa8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:24.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.385+0000 7fdaaeffd640 1 -- 172.21.15.186:0/2820675582 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fdab407b460 con 0x7fdab80caae0 2024-03-28T15:08:24.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.752+0000 7fdac03b5640 1 -- 172.21.15.186:0/2820675582 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdab80cfa70 con 0x7fdab80caae0 2024-03-28T15:08:24.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.752+0000 7fdaaeffd640 1 -- 172.21.15.186:0/2820675582 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6848 (secure 0 0 0) 0x7fdab407f760 con 0x7fdab80caae0 2024-03-28T15:08:24.753 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:24.753 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5137,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:24.381272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4790,"qdb_cluster":[4790]},"id":1}]} 2024-03-28T15:08:24.753 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 24 2024-03-28T15:08:24.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.755+0000 7fdac03b5640 1 -- 172.21.15.186:0/2820675582 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fda9c06c530 msgr2=0x7fda9c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:24.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.755+0000 7fdac03b5640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fda9c06c530 0x7fda9c06e9f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdab8151b80 tx=0x7fdaa8009190 comp rx=0 tx=0).stop 2024-03-28T15:08:24.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.755+0000 7fdac03b5640 1 -- 172.21.15.186:0/2820675582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdab80caae0 msgr2=0x7fdab8156400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:24.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.755+0000 7fdac03b5640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdab80caae0 0x7fdab8156400 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdab402b9e0 tx=0x7fdab4004730 comp rx=0 tx=0).stop 2024-03-28T15:08:24.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.755+0000 7fdac03b5640 1 -- 172.21.15.186:0/2820675582 shutdown_connections 2024-03-28T15:08:24.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.755+0000 7fdac03b5640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fda9c06c530 0x7fda9c06e9f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:24.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.755+0000 7fdac03b5640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdab8154970 0x7fdab814dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:24.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.755+0000 7fdac03b5640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdab80d0990 0x7fdab80c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:24.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.755+0000 7fdac03b5640 1 --2- 172.21.15.186:0/2820675582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdab80caae0 0x7fdab8156400 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:24.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.755+0000 7fdac03b5640 1 -- 172.21.15.186:0/2820675582 >> 172.21.15.186:0/2820675582 conn(0x7fdab8073400 msgr2=0x7fdab815a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:24.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.756+0000 7fdac03b5640 1 -- 172.21.15.186:0/2820675582 shutdown_connections 2024-03-28T15:08:24.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:24.756+0000 7fdac03b5640 1 -- 172.21.15.186:0/2820675582 wait complete. 2024-03-28T15:08:24.771 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:08:26.772 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:26.848 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:08:26.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:08:26.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.899+0000 7f32073e9640 1 Processor -- start 2024-03-28T15:08:26.899 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.899+0000 7f32073e9640 1 -- start start 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f32073e9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3200154bd0 0x7f320015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f32073e9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32000c6e00 con 0x7f32000c53c0 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f32073e9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f320015b080 con 0x7f32000c57e0 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f32073e9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f32000c5ca0 con 0x7f3200154bd0 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f3204984640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f32000c53c0 0x7f32000c66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:51664/0 (socket says 172.21.15.186:51664) 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f3204984640 1 -- 172.21.15.186:0/67818580 learned_addr learned my addr 172.21.15.186:0/67818580 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f3205185640 1 --2- 172.21.15.186:0/67818580 >> v2:172.21.15.191:3300/0 conn(0x7f3200154bd0 0x7f320015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f31f7fff640 1 -- 172.21.15.186:0/67818580 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2887002126 0 0) 0x7f32000c6e00 con 0x7f32000c53c0 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f31f7fff640 1 -- 172.21.15.186:0/67818580 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31ec003520 con 0x7f32000c53c0 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f3205185640 1 -- 172.21.15.186:0/67818580 >> v1:172.21.15.191:6789/0 conn(0x7f32000c57e0 legacy=0x7f32000cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f3205185640 1 -- 172.21.15.186:0/67818580 >> v1:172.21.15.186:6789/0 conn(0x7f32000c53c0 legacy=0x7f32000c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f3205185640 1 -- 172.21.15.186:0/67818580 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f320015b360 con 0x7f3200154bd0 2024-03-28T15:08:26.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.900+0000 7f31f7fff640 1 -- 172.21.15.186:0/67818580 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1959130936 0 0) 0x7f320015b080 con 0x7f32000c57e0 2024-03-28T15:08:26.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.901+0000 7f3205185640 1 --2- 172.21.15.186:0/67818580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3200154bd0 0x7f320015a7d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0097a0 tx=0x7f31fc02b9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec145b06287a2607 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:26.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.902+0000 7f31f7fff640 1 -- 172.21.15.186:0/67818580 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc02f800 con 0x7f3200154bd0 2024-03-28T15:08:26.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.902+0000 7f31f7fff640 1 -- 172.21.15.186:0/67818580 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31fc02f9a0 con 0x7f3200154bd0 2024-03-28T15:08:26.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.902+0000 7f31f7fff640 1 -- 172.21.15.186:0/67818580 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc02fd50 con 0x7f3200154bd0 2024-03-28T15:08:26.903 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.903+0000 7f32073e9640 1 -- 172.21.15.186:0/67818580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3200154bd0 msgr2=0x7f320015a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:26.903 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.903+0000 7f32073e9640 1 --2- 172.21.15.186:0/67818580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3200154bd0 0x7f320015a7d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0097a0 tx=0x7f31fc02b9e0 comp rx=0 tx=0).stop 2024-03-28T15:08:26.903 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.903+0000 7f32073e9640 1 -- 172.21.15.186:0/67818580 shutdown_connections 2024-03-28T15:08:26.903 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.903+0000 7f32073e9640 1 --2- 172.21.15.186:0/67818580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3200154bd0 0x7f320015a7d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:26.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.903+0000 7f32073e9640 1 -- 172.21.15.186:0/67818580 >> 172.21.15.186:0/67818580 conn(0x7f32000731f0 msgr2=0x7f3200073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:26.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.903+0000 7f32073e9640 1 -- 172.21.15.186:0/67818580 shutdown_connections 2024-03-28T15:08:26.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.903+0000 7f32073e9640 1 -- 172.21.15.186:0/67818580 wait complete. 2024-03-28T15:08:26.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.904+0000 7f32073e9640 1 Processor -- start 2024-03-28T15:08:26.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.904+0000 7f32073e9640 1 -- start start 2024-03-28T15:08:26.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f32073e9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32000c53c0 0x7f32001537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:26.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f32073e9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32000c57e0 0x7f32001e5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:26.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f32073e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3200154bd0 0x7f32001e5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:26.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f3205185640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32000c53c0 0x7f32001537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f3205185640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32000c53c0 0x7f32001537e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:50688/0 (socket says 172.21.15.186:50688) 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f3204984640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32000c57e0 0x7f32001e5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f3204984640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32000c57e0 0x7f32001e5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:55204/0 (socket says 172.21.15.186:55204) 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f3205185640 1 -- 172.21.15.186:0/1821023207 learned_addr learned my addr 172.21.15.186:0/1821023207 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f32000c61d0 con 0x7f3200154bd0 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f32000c5ef0 con 0x7f32000c53c0 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f32000cc390 con 0x7f32000c57e0 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.905+0000 7f3205986640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3200154bd0 0x7f32001e5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.906+0000 7f3205185640 1 -- 172.21.15.186:0/1821023207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32000c57e0 msgr2=0x7f32001e5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.906+0000 7f3205185640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32000c57e0 0x7f32001e5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.906+0000 7f3205185640 1 -- 172.21.15.186:0/1821023207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3200154bd0 msgr2=0x7f32001e5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.906+0000 7f3205185640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3200154bd0 0x7f32001e5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.906+0000 7f3205185640 1 -- 172.21.15.186:0/1821023207 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32001f10f0 con 0x7f32000c53c0 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.906+0000 7f3205185640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32000c53c0 0x7f32001537e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0098c0 tx=0x7f31fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:26.908 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.906+0000 7f31f5ffb640 1 -- 172.21.15.186:0/1821023207 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc037bc0 con 0x7f32000c53c0 2024-03-28T15:08:26.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.906+0000 7f31f5ffb640 1 -- 172.21.15.186:0/1821023207 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31fc03d040 con 0x7f32000c53c0 2024-03-28T15:08:26.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.906+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32000b4280 con 0x7f32000c53c0 2024-03-28T15:08:26.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.906+0000 7f31f5ffb640 1 -- 172.21.15.186:0/1821023207 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc043440 con 0x7f32000c53c0 2024-03-28T15:08:26.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.907+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32000b4840 con 0x7f32000c53c0 2024-03-28T15:08:26.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.908+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3200069500 con 0x7f32000c53c0 2024-03-28T15:08:26.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.909+0000 7f31f5ffb640 1 -- 172.21.15.186:0/1821023207 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f31fc0435e0 con 0x7f32000c53c0 2024-03-28T15:08:26.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.909+0000 7f31f5ffb640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f31e406c560 0x7f31e406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:26.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.910+0000 7f31f5ffb640 1 -- 172.21.15.186:0/1821023207 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f31fc0add90 con 0x7f32000c53c0 2024-03-28T15:08:26.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.910+0000 7f3204984640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f31e406c560 0x7f31e406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:26.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.910+0000 7f3204984640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f31e406c560 0x7f31e406ea20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f32000cf9a0 tx=0x7f31f0007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:26.915 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:26.915+0000 7f31f5ffb640 1 -- 172.21.15.186:0/1821023207 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f31fc051060 con 0x7f32000c53c0 2024-03-28T15:08:27.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.228+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32000cbb60 con 0x7f32000c53c0 2024-03-28T15:08:27.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.228+0000 7f31f5ffb640 1 -- 172.21.15.186:0/1821023207 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6848 (secure 0 0 0) 0x7f31fc033030 con 0x7f32000c53c0 2024-03-28T15:08:27.229 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:27.229 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5137,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:24.381272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4790,"qdb_cluster":[4790]},"id":1}]} 2024-03-28T15:08:27.229 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 24 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f31e406c560 msgr2=0x7f31e406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f31e406c560 0x7f31e406ea20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f32000cf9a0 tx=0x7f31f0007d30 comp rx=0 tx=0).stop 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32000c53c0 msgr2=0x7f32001537e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32000c53c0 0x7f32001537e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0098c0 tx=0x7f31fc0046c0 comp rx=0 tx=0).stop 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 shutdown_connections 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f31e406c560 0x7f31e406ea20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3200154bd0 0x7f32001e5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32000c57e0 0x7f32001e5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 --2- 172.21.15.186:0/1821023207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32000c53c0 0x7f32001537e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 >> 172.21.15.186:0/1821023207 conn(0x7f32000731f0 msgr2=0x7f32000cb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 shutdown_connections 2024-03-28T15:08:27.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:27.231+0000 7f32073e9640 1 -- 172.21.15.186:0/1821023207 wait complete. 2024-03-28T15:08:27.247 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:08:29.248 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:29.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.370+0000 7f92f1015640 1 Processor -- start 2024-03-28T15:08:29.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.370+0000 7f92f1015640 1 -- start start 2024-03-28T15:08:29.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92f1015640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f92ec0c53c0 0x7f92ec0c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:29.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92f1015640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c57e0 0x7f92ec0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:29.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92f1015640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92ec15b080 con 0x7f92ec154bd0 2024-03-28T15:08:29.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92f1015640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f92ec0c5ca0 con 0x7f92ec0c53c0 2024-03-28T15:08:29.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92f1015640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f92ec15b360 con 0x7f92ec0c57e0 2024-03-28T15:08:29.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92ea59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f92ec0c53c0 0x7f92ec0c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:29.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92ea59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f92ec0c53c0 0x7f92ec0c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43618/0 (socket says 172.21.15.186:43618) 2024-03-28T15:08:29.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92ead9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f92ec154bd0 0x7f92ec15a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:36044/0 (socket says 172.21.15.186:36044) 2024-03-28T15:08:29.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92ea59c640 1 -- 172.21.15.186:0/1705641649 learned_addr learned my addr 172.21.15.186:0/1705641649 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:29.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92eb59e640 1 --2- 172.21.15.186:0/1705641649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c57e0 0x7f92ec0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:29.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92ea59c640 1 -- 172.21.15.186:0/1705641649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c57e0 msgr2=0x7f92ec0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:29.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92ea59c640 1 --2- 172.21.15.186:0/1705641649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c57e0 0x7f92ec0cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:29.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92ea59c640 1 -- 172.21.15.186:0/1705641649 >> v1:172.21.15.186:6789/0 conn(0x7f92ec154bd0 legacy=0x7f92ec15a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:29.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.371+0000 7f92ea59c640 1 -- 172.21.15.186:0/1705641649 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ec15b640 con 0x7f92ec0c53c0 2024-03-28T15:08:29.373 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.372+0000 7f92ea59c640 1 --2- 172.21.15.186:0/1705641649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ec0c53c0 0x7f92ec0c66f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f92e0009870 tx=0x7f92e002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0ea848285d3f08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:29.373 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.372+0000 7f92e9d9b640 1 -- 172.21.15.186:0/1705641649 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e002f830 con 0x7f92ec0c53c0 2024-03-28T15:08:29.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.372+0000 7f92e9d9b640 1 -- 172.21.15.186:0/1705641649 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92e002f9d0 con 0x7f92ec0c53c0 2024-03-28T15:08:29.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.372+0000 7f92e9d9b640 1 -- 172.21.15.186:0/1705641649 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e002fda0 con 0x7f92ec0c53c0 2024-03-28T15:08:29.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.373+0000 7f92f1015640 1 -- 172.21.15.186:0/1705641649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ec0c53c0 msgr2=0x7f92ec0c66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:29.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.373+0000 7f92f1015640 1 --2- 172.21.15.186:0/1705641649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ec0c53c0 0x7f92ec0c66f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f92e0009870 tx=0x7f92e002ba10 comp rx=0 tx=0).stop 2024-03-28T15:08:29.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.373+0000 7f92f1015640 1 -- 172.21.15.186:0/1705641649 shutdown_connections 2024-03-28T15:08:29.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.373+0000 7f92f1015640 1 --2- 172.21.15.186:0/1705641649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c57e0 0x7f92ec0cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:29.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.373+0000 7f92f1015640 1 --2- 172.21.15.186:0/1705641649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ec0c53c0 0x7f92ec0c66f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:29.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.373+0000 7f92f1015640 1 -- 172.21.15.186:0/1705641649 >> 172.21.15.186:0/1705641649 conn(0x7f92ec0731f0 msgr2=0x7f92ec073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:29.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.373+0000 7f92f1015640 1 -- 172.21.15.186:0/1705641649 shutdown_connections 2024-03-28T15:08:29.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.373+0000 7f92f1015640 1 -- 172.21.15.186:0/1705641649 wait complete. 2024-03-28T15:08:29.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.375+0000 7f92f1015640 1 Processor -- start 2024-03-28T15:08:29.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.375+0000 7f92f1015640 1 -- start start 2024-03-28T15:08:29.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92f1015640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c53c0 0x7f92ec1f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92f1015640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ec0c57e0 0x7f92ec159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92f1015640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92ec154bd0 0x7f92ec06bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92f1015640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f92ec0cc390 con 0x7f92ec0c57e0 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92f1015640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f92ec0c61d0 con 0x7f92ec154bd0 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92f1015640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f92ec0c6c30 con 0x7f92ec0c53c0 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92ead9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c53c0 0x7f92ec1f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92ead9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c53c0 0x7f92ec1f4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:51938/0 (socket says 172.21.15.186:51938) 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92ead9d640 1 -- 172.21.15.186:0/923963936 learned_addr learned my addr 172.21.15.186:0/923963936 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92eb59e640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92ec154bd0 0x7f92ec06bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92ea59c640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ec0c57e0 0x7f92ec159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92ead9d640 1 -- 172.21.15.186:0/923963936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92ec154bd0 msgr2=0x7f92ec06bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:29.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92ead9d640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92ec154bd0 0x7f92ec06bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:29.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92ead9d640 1 -- 172.21.15.186:0/923963936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ec0c57e0 msgr2=0x7f92ec159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:29.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.376+0000 7f92ead9d640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ec0c57e0 0x7f92ec159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:29.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.377+0000 7f92ead9d640 1 -- 172.21.15.186:0/923963936 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ec15a4b0 con 0x7f92ec0c53c0 2024-03-28T15:08:29.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.377+0000 7f92ead9d640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c53c0 0x7f92ec1f4530 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f92dc00c430 tx=0x7f92dc00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:29.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.377+0000 7f92d37fe640 1 -- 172.21.15.186:0/923963936 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92dc0074e0 con 0x7f92ec0c53c0 2024-03-28T15:08:29.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.377+0000 7f92d37fe640 1 -- 172.21.15.186:0/923963936 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92dc007680 con 0x7f92ec0c53c0 2024-03-28T15:08:29.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.377+0000 7f92f1015640 1 -- 172.21.15.186:0/923963936 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92ec0ca4f0 con 0x7f92ec0c53c0 2024-03-28T15:08:29.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.377+0000 7f92d37fe640 1 -- 172.21.15.186:0/923963936 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92dc0049e0 con 0x7f92ec0c53c0 2024-03-28T15:08:29.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.377+0000 7f92f1015640 1 -- 172.21.15.186:0/923963936 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92ec158bc0 con 0x7f92ec0c53c0 2024-03-28T15:08:29.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.379+0000 7f92f1015640 1 -- 172.21.15.186:0/923963936 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92ec0cfd10 con 0x7f92ec0c53c0 2024-03-28T15:08:29.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.379+0000 7f92d37fe640 1 -- 172.21.15.186:0/923963936 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f92dc01a020 con 0x7f92ec0c53c0 2024-03-28T15:08:29.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.380+0000 7f92d37fe640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f92d806c560 0x7f92d806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:29.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.380+0000 7f92d37fe640 1 -- 172.21.15.186:0/923963936 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f92dc08afa0 con 0x7f92ec0c53c0 2024-03-28T15:08:29.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.380+0000 7f92ea59c640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f92d806c560 0x7f92d806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:29.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.380+0000 7f92ea59c640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f92d806c560 0x7f92d806ea20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f92e0036000 tx=0x7f92e0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:29.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.385+0000 7f92d37fe640 1 -- 172.21.15.186:0/923963936 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f92ec0cfd10 con 0x7f92ec0c53c0 2024-03-28T15:08:29.734 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.734+0000 7f92f1015640 1 -- 172.21.15.186:0/923963936 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92ec069500 con 0x7f92ec0c53c0 2024-03-28T15:08:29.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.734+0000 7f92d37fe640 1 -- 172.21.15.186:0/923963936 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6848 (secure 0 0 0) 0x7f92dc058d60 con 0x7f92ec0c53c0 2024-03-28T15:08:29.735 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:29.735 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5137,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:06:24.381272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6834/1992706523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1992706523},{"type":"v1","addr":"172.21.15.191:6834","nonce":1992706523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4790,"qdb_cluster":[4790]},"id":1}]} 2024-03-28T15:08:29.736 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 24 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 -- 172.21.15.186:0/923963936 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f92d806c560 msgr2=0x7f92d806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f92d806c560 0x7f92d806ea20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f92e0036000 tx=0x7f92e0008030 comp rx=0 tx=0).stop 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 -- 172.21.15.186:0/923963936 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c53c0 msgr2=0x7f92ec1f4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c53c0 0x7f92ec1f4530 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f92dc00c430 tx=0x7f92dc00c8f0 comp rx=0 tx=0).stop 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 -- 172.21.15.186:0/923963936 shutdown_connections 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f92d806c560 0x7f92d806ea20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92ec154bd0 0x7f92ec06bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ec0c57e0 0x7f92ec159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 --2- 172.21.15.186:0/923963936 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92ec0c53c0 0x7f92ec1f4530 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 -- 172.21.15.186:0/923963936 >> 172.21.15.186:0/923963936 conn(0x7f92ec0731f0 msgr2=0x7f92ec153a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 -- 172.21.15.186:0/923963936 shutdown_connections 2024-03-28T15:08:29.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:29.737+0000 7f92f1015640 1 -- 172.21.15.186:0/923963936 wait complete. 2024-03-28T15:08:29.750 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T15:08:31.751 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:31.875 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.874+0000 7fe998671640 1 Processor -- start 2024-03-28T15:08:31.875 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.874+0000 7fe998671640 1 -- start start 2024-03-28T15:08:31.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.875+0000 7fe998671640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe9900c53c0 0x7fe9900c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:31.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.875+0000 7fe998671640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe9900c57e0 0x7fe9900cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:31.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.875+0000 7fe998671640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe99015b080 con 0x7fe990154bd0 2024-03-28T15:08:31.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.875+0000 7fe998671640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fe9900c5ca0 con 0x7fe9900c57e0 2024-03-28T15:08:31.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.875+0000 7fe998671640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fe99015b360 con 0x7fe9900c53c0 2024-03-28T15:08:31.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.876+0000 7fe996c0e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe9900c57e0 0x7fe9900cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:31.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.876+0000 7fe996c0e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe9900c57e0 0x7fe9900cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43628/0 (socket says 172.21.15.186:43628) 2024-03-28T15:08:31.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.876+0000 7fe996c0e640 1 -- 172.21.15.186:0/1554715011 learned_addr learned my addr 172.21.15.186:0/1554715011 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:31.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.876+0000 7fe995c0c640 1 --2- 172.21.15.186:0/1554715011 >> v2:172.21.15.191:3300/0 conn(0x7fe9900c53c0 0x7fe9900c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:31.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.876+0000 7fe996c0e640 1 -- 172.21.15.186:0/1554715011 >> v2:172.21.15.191:3300/0 conn(0x7fe9900c53c0 msgr2=0x7fe9900c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:31.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.876+0000 7fe996c0e640 1 --2- 172.21.15.186:0/1554715011 >> v2:172.21.15.191:3300/0 conn(0x7fe9900c53c0 0x7fe9900c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:31.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.876+0000 7fe996c0e640 1 -- 172.21.15.186:0/1554715011 >> v1:172.21.15.191:6789/0 conn(0x7fe990154bd0 legacy=0x7fe99015a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:31.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.876+0000 7fe996c0e640 1 -- 172.21.15.186:0/1554715011 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99015b640 con 0x7fe9900c57e0 2024-03-28T15:08:31.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.876+0000 7fe995c0c640 1 --2- 172.21.15.186:0/1554715011 >> v2:172.21.15.191:3300/0 conn(0x7fe9900c53c0 0x7fe9900c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:08:31.878 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.877+0000 7fe996c0e640 1 --2- 172.21.15.186:0/1554715011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9900c57e0 0x7fe9900cb040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe99008d6b0 tx=0x7fe98802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec2ddb5bbc8aa3a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:31.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.877+0000 7fe99540b640 1 -- 172.21.15.186:0/1554715011 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98802cb20 con 0x7fe9900c57e0 2024-03-28T15:08:31.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.877+0000 7fe99540b640 1 -- 172.21.15.186:0/1554715011 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe98802ccc0 con 0x7fe9900c57e0 2024-03-28T15:08:31.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.877+0000 7fe99540b640 1 -- 172.21.15.186:0/1554715011 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988034610 con 0x7fe9900c57e0 2024-03-28T15:08:31.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.877+0000 7fe998671640 1 -- 172.21.15.186:0/1554715011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9900c57e0 msgr2=0x7fe9900cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:31.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.878+0000 7fe998671640 1 --2- 172.21.15.186:0/1554715011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9900c57e0 0x7fe9900cb040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe99008d6b0 tx=0x7fe98802bf80 comp rx=0 tx=0).stop 2024-03-28T15:08:31.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.878+0000 7fe998671640 1 -- 172.21.15.186:0/1554715011 shutdown_connections 2024-03-28T15:08:31.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.878+0000 7fe998671640 1 --2- 172.21.15.186:0/1554715011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9900c57e0 0x7fe9900cb040 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:31.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.878+0000 7fe998671640 1 --2- 172.21.15.186:0/1554715011 >> v2:172.21.15.191:3300/0 conn(0x7fe9900c53c0 0x7fe9900c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:31.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.878+0000 7fe998671640 1 -- 172.21.15.186:0/1554715011 >> 172.21.15.186:0/1554715011 conn(0x7fe9900731f0 msgr2=0x7fe990073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:31.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.878+0000 7fe998671640 1 -- 172.21.15.186:0/1554715011 shutdown_connections 2024-03-28T15:08:31.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.878+0000 7fe998671640 1 -- 172.21.15.186:0/1554715011 wait complete. 2024-03-28T15:08:31.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.879+0000 7fe998671640 1 Processor -- start 2024-03-28T15:08:31.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.879+0000 7fe998671640 1 -- start start 2024-03-28T15:08:31.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe998671640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9900c53c0 0x7fe990150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:31.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe998671640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9900c57e0 0x7fe9901f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:31.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe998671640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe990154bd0 0x7fe99014aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:31.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe998671640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe9900cc390 con 0x7fe990154bd0 2024-03-28T15:08:31.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe998671640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe9900c61d0 con 0x7fe9900c57e0 2024-03-28T15:08:31.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe998671640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fe9900c6c30 con 0x7fe9900c53c0 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe996c0e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe990154bd0 0x7fe99014aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe996c0e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe990154bd0 0x7fe99014aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43644/0 (socket says 172.21.15.186:43644) 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe996c0e640 1 -- 172.21.15.186:0/2260235791 learned_addr learned my addr 172.21.15.186:0/2260235791 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe995c0c640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9900c57e0 0x7fe9901f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe99640d640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9900c53c0 0x7fe990150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe996c0e640 1 -- 172.21.15.186:0/2260235791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9900c53c0 msgr2=0x7fe990150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe996c0e640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9900c53c0 0x7fe990150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe996c0e640 1 -- 172.21.15.186:0/2260235791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9900c57e0 msgr2=0x7fe9901f11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe996c0e640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9900c57e0 0x7fe9901f11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe996c0e640 1 -- 172.21.15.186:0/2260235791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9901f4400 con 0x7fe990154bd0 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe995c0c640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9900c57e0 0x7fe9901f11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe99640d640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9900c53c0 0x7fe990150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe996c0e640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe990154bd0 0x7fe99014aa70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe988009f40 tx=0x7fe988004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:31.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.880+0000 7fe986ffd640 1 -- 172.21.15.186:0/2260235791 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988043020 con 0x7fe990154bd0 2024-03-28T15:08:31.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.881+0000 7fe986ffd640 1 -- 172.21.15.186:0/2260235791 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe98803e450 con 0x7fe990154bd0 2024-03-28T15:08:31.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.881+0000 7fe998671640 1 -- 172.21.15.186:0/2260235791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9901f46e0 con 0x7fe990154bd0 2024-03-28T15:08:31.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.881+0000 7fe986ffd640 1 -- 172.21.15.186:0/2260235791 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988009510 con 0x7fe990154bd0 2024-03-28T15:08:31.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.881+0000 7fe998671640 1 -- 172.21.15.186:0/2260235791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9901f4ca0 con 0x7fe990154bd0 2024-03-28T15:08:31.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.882+0000 7fe998671640 1 -- 172.21.15.186:0/2260235791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9900b3a70 con 0x7fe990154bd0 2024-03-28T15:08:31.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.883+0000 7fe986ffd640 1 -- 172.21.15.186:0/2260235791 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe988046020 con 0x7fe990154bd0 2024-03-28T15:08:31.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.883+0000 7fe986ffd640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe97406c560 0x7fe97406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:31.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.883+0000 7fe986ffd640 1 -- 172.21.15.186:0/2260235791 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7fe9880ade00 con 0x7fe990154bd0 2024-03-28T15:08:31.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.883+0000 7fe99640d640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe97406c560 0x7fe97406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:31.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.884+0000 7fe99640d640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe97406c560 0x7fe97406ea20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe98c003eb0 tx=0x7fe98c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:31.889 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:31.888+0000 7fe986ffd640 1 -- 172.21.15.186:0/2260235791 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fe9900b3a70 con 0x7fe990154bd0 2024-03-28T15:08:32.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.206+0000 7fe998671640 1 -- 172.21.15.186:0/2260235791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe990069500 con 0x7fe990154bd0 2024-03-28T15:08:32.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.207+0000 7fe986ffd640 1 -- 172.21.15.186:0/2260235791 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6012 (secure 0 0 0) 0x7fe98804d070 con 0x7fe990154bd0 2024-03-28T15:08:32.207 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:32.207 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:31.221047+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:08:32.208 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 25 2024-03-28T15:08:32.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.209+0000 7fe998671640 1 -- 172.21.15.186:0/2260235791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe97406c560 msgr2=0x7fe97406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:32.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.209+0000 7fe998671640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe97406c560 0x7fe97406ea20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe98c003eb0 tx=0x7fe98c009040 comp rx=0 tx=0).stop 2024-03-28T15:08:32.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.209+0000 7fe998671640 1 -- 172.21.15.186:0/2260235791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe990154bd0 msgr2=0x7fe99014aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:32.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.209+0000 7fe998671640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe990154bd0 0x7fe99014aa70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe988009f40 tx=0x7fe988004a40 comp rx=0 tx=0).stop 2024-03-28T15:08:32.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.210+0000 7fe998671640 1 -- 172.21.15.186:0/2260235791 shutdown_connections 2024-03-28T15:08:32.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.210+0000 7fe998671640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fe97406c560 0x7fe97406ea20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.210+0000 7fe998671640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe990154bd0 0x7fe99014aa70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.210+0000 7fe998671640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9900c57e0 0x7fe9901f11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.210+0000 7fe998671640 1 --2- 172.21.15.186:0/2260235791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9900c53c0 0x7fe990150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.210+0000 7fe998671640 1 -- 172.21.15.186:0/2260235791 >> 172.21.15.186:0/2260235791 conn(0x7fe9900731f0 msgr2=0x7fe9900cb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:32.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.210+0000 7fe998671640 1 -- 172.21.15.186:0/2260235791 shutdown_connections 2024-03-28T15:08:32.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.210+0000 7fe998671640 1 -- 172.21.15.186:0/2260235791 wait complete. 2024-03-28T15:08:32.225 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-03-28T15:08:32.225 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-28T15:08:32.225 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:32.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.359+0000 7fb105ca4640 1 Processor -- start 2024-03-28T15:08:32.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.360+0000 7fb105ca4640 1 -- start start 2024-03-28T15:08:32.362 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.361+0000 7fb105ca4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1000c3f10 0x7fb1000735a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:32.362 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.361+0000 7fb105ca4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb100073ae0 0x7fb100157730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.361+0000 7fb105ca4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1000c84c0 con 0x7fb100151930 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.361+0000 7fb105ca4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fb1000c9240 con 0x7fb100073ae0 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.361+0000 7fb105ca4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb10006c870 con 0x7fb1000c3f10 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.361+0000 7fb0fffff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb100073ae0 0x7fb100157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.361+0000 7fb0fffff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb100073ae0 0x7fb100157730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:47908/0 (socket says 172.21.15.186:47908) 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.361+0000 7fb0fffff640 1 -- 172.21.15.186:0/2464978257 learned_addr learned my addr 172.21.15.186:0/2464978257 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.361+0000 7fb0feffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1000c3f10 0x7fb1000735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.362+0000 7fb0fe7fc640 1 -- 172.21.15.186:0/2464978257 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 781426290 0 0) 0x7fb1000c84c0 con 0x7fb100151930 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.362+0000 7fb0fe7fc640 1 -- 172.21.15.186:0/2464978257 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0e8005730 con 0x7fb100151930 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.362+0000 7fb0fffff640 1 -- 172.21.15.186:0/2464978257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1000c3f10 msgr2=0x7fb1000735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.362+0000 7fb0fffff640 1 --2- 172.21.15.186:0/2464978257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1000c3f10 0x7fb1000735a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.362+0000 7fb0fffff640 1 -- 172.21.15.186:0/2464978257 >> v1:172.21.15.191:6789/0 conn(0x7fb100151930 legacy=0x7fb100072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:32.363 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.362+0000 7fb0fffff640 1 -- 172.21.15.186:0/2464978257 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10006cac0 con 0x7fb100073ae0 2024-03-28T15:08:32.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.363+0000 7fb0fffff640 1 --2- 172.21.15.186:0/2464978257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb100073ae0 0x7fb100157730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8009e50 tx=0x7fb0f802bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55b3090b6362de54 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:32.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.363+0000 7fb0fe7fc640 1 -- 172.21.15.186:0/2464978257 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f802cb90 con 0x7fb100073ae0 2024-03-28T15:08:32.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.363+0000 7fb0fe7fc640 1 -- 172.21.15.186:0/2464978257 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f802cd30 con 0x7fb100073ae0 2024-03-28T15:08:32.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.363+0000 7fb0fe7fc640 1 -- 172.21.15.186:0/2464978257 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8034610 con 0x7fb100073ae0 2024-03-28T15:08:32.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.364+0000 7fb105ca4640 1 -- 172.21.15.186:0/2464978257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb100073ae0 msgr2=0x7fb100157730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:32.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.364+0000 7fb105ca4640 1 --2- 172.21.15.186:0/2464978257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb100073ae0 0x7fb100157730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8009e50 tx=0x7fb0f802bff0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.364+0000 7fb105ca4640 1 -- 172.21.15.186:0/2464978257 shutdown_connections 2024-03-28T15:08:32.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.364+0000 7fb105ca4640 1 --2- 172.21.15.186:0/2464978257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb100073ae0 0x7fb100157730 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.364+0000 7fb105ca4640 1 --2- 172.21.15.186:0/2464978257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1000c3f10 0x7fb1000735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.364+0000 7fb105ca4640 1 -- 172.21.15.186:0/2464978257 >> 172.21.15.186:0/2464978257 conn(0x7fb100070020 msgr2=0x7fb100070400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:32.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.364+0000 7fb105ca4640 1 -- 172.21.15.186:0/2464978257 shutdown_connections 2024-03-28T15:08:32.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.364+0000 7fb105ca4640 1 -- 172.21.15.186:0/2464978257 wait complete. 2024-03-28T15:08:32.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.366+0000 7fb105ca4640 1 Processor -- start 2024-03-28T15:08:32.366 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.366+0000 7fb105ca4640 1 -- start start 2024-03-28T15:08:32.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.367+0000 7fb105ca4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb100073ae0 0x7fb100144790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:32.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.367+0000 7fb105ca4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1000c3f10 0x7fb10014cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:32.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.367+0000 7fb105ca4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb100151930 0x7fb1001427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.367+0000 7fb105ca4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1000c8d20 con 0x7fb100073ae0 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.367+0000 7fb105ca4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb1000c8a40 con 0x7fb100151930 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.367+0000 7fb105ca4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb10006cda0 con 0x7fb1000c3f10 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.367+0000 7fb0ff7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb100073ae0 0x7fb100144790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.367+0000 7fb0ff7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb100073ae0 0x7fb100144790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43656/0 (socket says 172.21.15.186:43656) 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.367+0000 7fb0feffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1000c3f10 0x7fb10014cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.367+0000 7fb0ff7fe640 1 -- 172.21.15.186:0/1134264790 learned_addr learned my addr 172.21.15.186:0/1134264790 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.368+0000 7fb0fffff640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb100151930 0x7fb1001427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.368+0000 7fb0feffd640 1 -- 172.21.15.186:0/1134264790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb100151930 msgr2=0x7fb1001427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.368+0000 7fb0feffd640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb100151930 0x7fb1001427a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.368+0000 7fb0feffd640 1 -- 172.21.15.186:0/1134264790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb100073ae0 msgr2=0x7fb100144790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.368+0000 7fb0feffd640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb100073ae0 0x7fb100144790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.368+0000 7fb0feffd640 1 -- 172.21.15.186:0/1134264790 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb100159190 con 0x7fb1000c3f10 2024-03-28T15:08:32.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.368+0000 7fb0feffd640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1000c3f10 0x7fb10014cfe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400e820 tx=0x7fb0f400ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:32.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.368+0000 7fb0e7fff640 1 -- 172.21.15.186:0/1134264790 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4017070 con 0x7fb1000c3f10 2024-03-28T15:08:32.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.368+0000 7fb0e7fff640 1 -- 172.21.15.186:0/1134264790 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f400c940 con 0x7fb1000c3f10 2024-03-28T15:08:32.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.368+0000 7fb0e7fff640 1 -- 172.21.15.186:0/1134264790 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f400cd10 con 0x7fb1000c3f10 2024-03-28T15:08:32.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.368+0000 7fb105ca4640 1 -- 172.21.15.186:0/1134264790 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10015a4d0 con 0x7fb1000c3f10 2024-03-28T15:08:32.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.369+0000 7fb105ca4640 1 -- 172.21.15.186:0/1134264790 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb100159710 con 0x7fb1000c3f10 2024-03-28T15:08:32.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.370+0000 7fb105ca4640 1 -- 172.21.15.186:0/1134264790 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1000686d0 con 0x7fb1000c3f10 2024-03-28T15:08:32.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.371+0000 7fb0e7fff640 1 -- 172.21.15.186:0/1134264790 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb0f401a070 con 0x7fb1000c3f10 2024-03-28T15:08:32.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.371+0000 7fb0e7fff640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0ec06c530 0x7fb0ec06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:32.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.372+0000 7fb0e7fff640 1 -- 172.21.15.186:0/1134264790 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7fb0f408b700 con 0x7fb1000c3f10 2024-03-28T15:08:32.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.372+0000 7fb0ff7fe640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0ec06c530 0x7fb0ec06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:32.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.372+0000 7fb0ff7fe640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0ec06c530 0x7fb0ec06e9f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0007fd0 tx=0x7fb0f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:32.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.376+0000 7fb0e7fff640 1 -- 172.21.15.186:0/1134264790 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb0f4059500 con 0x7fb1000c3f10 2024-03-28T15:08:32.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.743+0000 7fb105ca4640 1 -- 172.21.15.186:0/1134264790 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb100146380 con 0x7fb1000c3f10 2024-03-28T15:08:32.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.744+0000 7fb0e7fff640 1 -- 172.21.15.186:0/1134264790 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6012 (secure 0 0 0) 0x7fb0f405d800 con 0x7fb1000c3f10 2024-03-28T15:08:32.744 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:32.744 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:31.221047+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:08:32.744 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 25 2024-03-28T15:08:32.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.746+0000 7fb105ca4640 1 -- 172.21.15.186:0/1134264790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0ec06c530 msgr2=0x7fb0ec06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:32.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.746+0000 7fb105ca4640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0ec06c530 0x7fb0ec06e9f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0007fd0 tx=0x7fb0f0009040 comp rx=0 tx=0).stop 2024-03-28T15:08:32.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.746+0000 7fb105ca4640 1 -- 172.21.15.186:0/1134264790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1000c3f10 msgr2=0x7fb10014cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:32.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.746+0000 7fb105ca4640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1000c3f10 0x7fb10014cfe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400e820 tx=0x7fb0f400ece0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.747+0000 7fb105ca4640 1 -- 172.21.15.186:0/1134264790 shutdown_connections 2024-03-28T15:08:32.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.747+0000 7fb105ca4640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb0ec06c530 0x7fb0ec06e9f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.747+0000 7fb105ca4640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb100151930 0x7fb1001427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.747+0000 7fb105ca4640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1000c3f10 0x7fb10014cfe0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.747+0000 7fb105ca4640 1 --2- 172.21.15.186:0/1134264790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb100073ae0 0x7fb100144790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:32.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.747+0000 7fb105ca4640 1 -- 172.21.15.186:0/1134264790 >> 172.21.15.186:0/1134264790 conn(0x7fb100070020 msgr2=0x7fb100067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:32.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.747+0000 7fb105ca4640 1 -- 172.21.15.186:0/1134264790 shutdown_connections 2024-03-28T15:08:32.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:32.747+0000 7fb105ca4640 1 -- 172.21.15.186:0/1134264790 wait complete. 2024-03-28T15:08:32.762 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T15:01:31.425299+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5137": { "addr": "172.21.15.191:6837/3037586083", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3037586083, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3037586083, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5137, "incarnation": 25, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T15:08:31.221047+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5137 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/1278716016", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6833", "nonce": 1278716016, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1278716016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6835/2068761001", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2068761001, "type": "v2" }, { "addr": "172.21.15.186:6835", "nonce": 2068761001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2344737463", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6837", "nonce": 2344737463, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2344737463, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4284, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/1548903381", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 1548903381, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 1548903381, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T15:08:34.763 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:34.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.909+0000 7f7aefa37640 1 Processor -- start 2024-03-28T15:08:34.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.909+0000 7f7aefa37640 1 -- start start 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aefa37640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7ae8154bd0 0x7f7ae815a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aefa37640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7ae80c53c0 0x7f7ae80c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aefa37640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae815b080 con 0x7f7ae80c57e0 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aefa37640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f7ae80c5ca0 con 0x7f7ae80c53c0 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aefa37640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f7ae815b360 con 0x7f7ae8154bd0 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aecfd2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7ae80c53c0 0x7f7ae80c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aecfd2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7ae80c53c0 0x7f7ae80c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43672/0 (socket says 172.21.15.186:43672) 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aecfd2640 1 -- 172.21.15.186:0/2379053326 learned_addr learned my addr 172.21.15.186:0/2379053326 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aed7d3640 1 --2- 172.21.15.186:0/2379053326 >> v2:172.21.15.191:3300/0 conn(0x7f7ae8154bd0 0x7f7ae815a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aecfd2640 1 -- 172.21.15.186:0/2379053326 >> v2:172.21.15.191:3300/0 conn(0x7f7ae8154bd0 msgr2=0x7f7ae815a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aecfd2640 1 --2- 172.21.15.186:0/2379053326 >> v2:172.21.15.191:3300/0 conn(0x7f7ae8154bd0 0x7f7ae815a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:34.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aecfd2640 1 -- 172.21.15.186:0/2379053326 >> v1:172.21.15.191:6789/0 conn(0x7f7ae80c57e0 legacy=0x7f7ae80cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:34.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.910+0000 7f7aecfd2640 1 -- 172.21.15.186:0/2379053326 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae815b640 con 0x7f7ae80c53c0 2024-03-28T15:08:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.911+0000 7f7aecfd2640 1 --2- 172.21.15.186:0/2379053326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae80c53c0 0x7f7ae80c66f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80d0b90 tx=0x7f7ad802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82f49cb10719382 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.911+0000 7f7adffff640 1 -- 172.21.15.186:0/2379053326 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad802f850 con 0x7f7ae80c53c0 2024-03-28T15:08:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.911+0000 7f7adffff640 1 -- 172.21.15.186:0/2379053326 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ad802f9f0 con 0x7f7ae80c53c0 2024-03-28T15:08:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.911+0000 7f7adffff640 1 -- 172.21.15.186:0/2379053326 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad802fdc0 con 0x7f7ae80c53c0 2024-03-28T15:08:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.912+0000 7f7aefa37640 1 -- 172.21.15.186:0/2379053326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae80c53c0 msgr2=0x7f7ae80c66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.912+0000 7f7aefa37640 1 --2- 172.21.15.186:0/2379053326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae80c53c0 0x7f7ae80c66f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80d0b90 tx=0x7f7ad802ba30 comp rx=0 tx=0).stop 2024-03-28T15:08:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.912+0000 7f7aefa37640 1 -- 172.21.15.186:0/2379053326 shutdown_connections 2024-03-28T15:08:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.912+0000 7f7aefa37640 1 --2- 172.21.15.186:0/2379053326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae80c53c0 0x7f7ae80c66f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.912+0000 7f7aefa37640 1 --2- 172.21.15.186:0/2379053326 >> v2:172.21.15.191:3300/0 conn(0x7f7ae8154bd0 0x7f7ae815a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.912+0000 7f7aefa37640 1 -- 172.21.15.186:0/2379053326 >> 172.21.15.186:0/2379053326 conn(0x7f7ae80731f0 msgr2=0x7f7ae8073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:34.914 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.912+0000 7f7aefa37640 1 -- 172.21.15.186:0/2379053326 shutdown_connections 2024-03-28T15:08:34.914 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.912+0000 7f7aefa37640 1 -- 172.21.15.186:0/2379053326 wait complete. 2024-03-28T15:08:34.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.913+0000 7f7aefa37640 1 Processor -- start 2024-03-28T15:08:34.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.913+0000 7f7aefa37640 1 -- start start 2024-03-28T15:08:34.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.914+0000 7f7aefa37640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7ae80c53c0 0x7f7ae8157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:34.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.914+0000 7f7aefa37640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae80c57e0 0x7f7ae81e7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:34.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.914+0000 7f7aefa37640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ae8154bd0 0x7f7ae814aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:34.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.914+0000 7f7aefa37640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae80cc390 con 0x7f7ae80c57e0 2024-03-28T15:08:34.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.914+0000 7f7aefa37640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae80c61d0 con 0x7f7ae8154bd0 2024-03-28T15:08:34.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.914+0000 7f7aefa37640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7ae815ad10 con 0x7f7ae80c53c0 2024-03-28T15:08:34.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.914+0000 7f7aecfd2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae80c57e0 0x7f7ae81e7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.914+0000 7f7aecfd2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae80c57e0 0x7f7ae81e7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43686/0 (socket says 172.21.15.186:43686) 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.914+0000 7f7aecfd2640 1 -- 172.21.15.186:0/3006873923 learned_addr learned my addr 172.21.15.186:0/3006873923 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.914+0000 7f7aed7d3640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7ae80c53c0 0x7f7ae8157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.914+0000 7f7aedfd4640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ae8154bd0 0x7f7ae814aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.915+0000 7f7aecfd2640 1 -- 172.21.15.186:0/3006873923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7ae80c53c0 msgr2=0x7f7ae8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.915+0000 7f7aecfd2640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7ae80c53c0 0x7f7ae8157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.915+0000 7f7aecfd2640 1 -- 172.21.15.186:0/3006873923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ae8154bd0 msgr2=0x7f7ae814aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.915+0000 7f7aecfd2640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ae8154bd0 0x7f7ae814aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.915+0000 7f7aecfd2640 1 -- 172.21.15.186:0/3006873923 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae81f1350 con 0x7f7ae80c57e0 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.915+0000 7f7aecfd2640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae80c57e0 0x7f7ae81e7520 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8000c00 tx=0x7f7ad8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.915+0000 7f7addffb640 1 -- 172.21.15.186:0/3006873923 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8047070 con 0x7f7ae80c57e0 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.915+0000 7f7aefa37640 1 -- 172.21.15.186:0/3006873923 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae81f36a0 con 0x7f7ae80c57e0 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.915+0000 7f7addffb640 1 -- 172.21.15.186:0/3006873923 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ad8037d30 con 0x7f7ae80c57e0 2024-03-28T15:08:34.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.915+0000 7f7addffb640 1 -- 172.21.15.186:0/3006873923 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad80426c0 con 0x7f7ae80c57e0 2024-03-28T15:08:34.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.915+0000 7f7aefa37640 1 -- 172.21.15.186:0/3006873923 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae81f3c60 con 0x7f7ae80c57e0 2024-03-28T15:08:34.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.917+0000 7f7aefa37640 1 -- 172.21.15.186:0/3006873923 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ae8069500 con 0x7f7ae80c57e0 2024-03-28T15:08:34.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.918+0000 7f7addffb640 1 -- 172.21.15.186:0/3006873923 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7ad804a070 con 0x7f7ae80c57e0 2024-03-28T15:08:34.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.918+0000 7f7addffb640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7acc06c530 0x7f7acc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:34.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.918+0000 7f7addffb640 1 -- 172.21.15.186:0/3006873923 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7f7ad80adb10 con 0x7f7ae80c57e0 2024-03-28T15:08:34.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.919+0000 7f7aed7d3640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7acc06c530 0x7f7acc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:34.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.919+0000 7f7aed7d3640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7acc06c530 0x7f7acc06e9f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009af0 tx=0x7f7ae4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:34.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:34.923+0000 7f7addffb640 1 -- 172.21.15.186:0/3006873923 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7ad8033030 con 0x7f7ae80c57e0 2024-03-28T15:08:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.283+0000 7f7aefa37640 1 -- 172.21.15.186:0/3006873923 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ae81543f0 con 0x7f7ae80c57e0 2024-03-28T15:08:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.284+0000 7f7addffb640 1 -- 172.21.15.186:0/3006873923 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6012 (secure 0 0 0) 0x7f7ad807b9a0 con 0x7f7ae80c57e0 2024-03-28T15:08:35.284 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:35.284 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:31.221047+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:08:35.285 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 25 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 -- 172.21.15.186:0/3006873923 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7acc06c530 msgr2=0x7f7acc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7acc06c530 0x7f7acc06e9f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009af0 tx=0x7f7ae4009190 comp rx=0 tx=0).stop 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 -- 172.21.15.186:0/3006873923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae80c57e0 msgr2=0x7f7ae81e7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae80c57e0 0x7f7ae81e7520 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8000c00 tx=0x7f7ad8004730 comp rx=0 tx=0).stop 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 -- 172.21.15.186:0/3006873923 shutdown_connections 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f7acc06c530 0x7f7acc06e9f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ae8154bd0 0x7f7ae814aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ae80c57e0 0x7f7ae81e7520 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 --2- 172.21.15.186:0/3006873923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7ae80c53c0 0x7f7ae8157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 -- 172.21.15.186:0/3006873923 >> 172.21.15.186:0/3006873923 conn(0x7f7ae80731f0 msgr2=0x7f7ae80ce030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 -- 172.21.15.186:0/3006873923 shutdown_connections 2024-03-28T15:08:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:35.286+0000 7f7aefa37640 1 -- 172.21.15.186:0/3006873923 wait complete. 2024-03-28T15:08:35.301 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:08:37.303 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:37.427 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.426+0000 7fd335b9a640 1 Processor -- start 2024-03-28T15:08:37.427 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.426+0000 7fd335b9a640 1 -- start start 2024-03-28T15:08:37.427 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.427+0000 7fd335b9a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd330159000 0x7fd3300c5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:37.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.427+0000 7fd335b9a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd3300c5590 0x7fd3300cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:37.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.427+0000 7fd335b9a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd33015ecf0 con 0x7fd3300b2ec0 2024-03-28T15:08:37.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.427+0000 7fd335b9a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd3300d0e90 con 0x7fd330159000 2024-03-28T15:08:37.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.427+0000 7fd335b9a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fd33015ef90 con 0x7fd3300c5590 2024-03-28T15:08:37.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.427+0000 7fd32f7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd330159000 0x7fd3300c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:37.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.427+0000 7fd32ffff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd3300b2ec0 0x7fd3301621d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:50126/0 (socket says 172.21.15.186:50126) 2024-03-28T15:08:37.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.427+0000 7fd32ffff640 1 -- 172.21.15.186:0/2753930483 learned_addr learned my addr 172.21.15.186:0/2753930483 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:08:37.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.427+0000 7fd32effd640 1 --2- 172.21.15.186:0/2753930483 >> v2:172.21.15.191:3300/0 conn(0x7fd3300c5590 0x7fd3300cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:37.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.428+0000 7fd32e7fc640 1 -- 172.21.15.186:0/2753930483 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2789266628 0 0) 0x7fd33015ecf0 con 0x7fd3300b2ec0 2024-03-28T15:08:37.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.428+0000 7fd32e7fc640 1 -- 172.21.15.186:0/2753930483 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd318003520 con 0x7fd3300b2ec0 2024-03-28T15:08:37.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.428+0000 7fd32effd640 1 -- 172.21.15.186:0/2753930483 >> v2:172.21.15.186:3300/0 conn(0x7fd330159000 msgr2=0x7fd3300c5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:37.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.428+0000 7fd32effd640 1 --2- 172.21.15.186:0/2753930483 >> v2:172.21.15.186:3300/0 conn(0x7fd330159000 0x7fd3300c5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:37.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.428+0000 7fd32effd640 1 -- 172.21.15.186:0/2753930483 >> v1:172.21.15.191:6789/0 conn(0x7fd3300b2ec0 legacy=0x7fd3301621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:37.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.428+0000 7fd32effd640 1 -- 172.21.15.186:0/2753930483 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33015f270 con 0x7fd3300c5590 2024-03-28T15:08:37.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.428+0000 7fd32f7fe640 1 --2- 172.21.15.186:0/2753930483 >> v2:172.21.15.186:3300/0 conn(0x7fd330159000 0x7fd3300c5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:08:37.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.429+0000 7fd32effd640 1 --2- 172.21.15.186:0/2753930483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c5590 0x7fd3300cbff0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd33008d910 tx=0x7fd32402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7125add4fd16c19 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:37.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.429+0000 7fd32e7fc640 1 -- 172.21.15.186:0/2753930483 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324034070 con 0x7fd3300c5590 2024-03-28T15:08:37.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.429+0000 7fd32e7fc640 1 -- 172.21.15.186:0/2753930483 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd324030610 con 0x7fd3300c5590 2024-03-28T15:08:37.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.429+0000 7fd32e7fc640 1 -- 172.21.15.186:0/2753930483 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3240309e0 con 0x7fd3300c5590 2024-03-28T15:08:37.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.430+0000 7fd335b9a640 1 -- 172.21.15.186:0/2753930483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c5590 msgr2=0x7fd3300cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:37.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.430+0000 7fd335b9a640 1 --2- 172.21.15.186:0/2753930483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c5590 0x7fd3300cbff0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd33008d910 tx=0x7fd32402b960 comp rx=0 tx=0).stop 2024-03-28T15:08:37.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.430+0000 7fd335b9a640 1 -- 172.21.15.186:0/2753930483 shutdown_connections 2024-03-28T15:08:37.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.430+0000 7fd335b9a640 1 --2- 172.21.15.186:0/2753930483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c5590 0x7fd3300cbff0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.430+0000 7fd335b9a640 1 --2- 172.21.15.186:0/2753930483 >> v2:172.21.15.186:3300/0 conn(0x7fd330159000 0x7fd3300c5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.430+0000 7fd335b9a640 1 -- 172.21.15.186:0/2753930483 >> 172.21.15.186:0/2753930483 conn(0x7fd3300731f0 msgr2=0x7fd330073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.430+0000 7fd335b9a640 1 -- 172.21.15.186:0/2753930483 shutdown_connections 2024-03-28T15:08:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.430+0000 7fd335b9a640 1 -- 172.21.15.186:0/2753930483 wait complete. 2024-03-28T15:08:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.431+0000 7fd335b9a640 1 Processor -- start 2024-03-28T15:08:37.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.431+0000 7fd335b9a640 1 -- start start 2024-03-28T15:08:37.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd335b9a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3300b2ec0 0x7fd3301579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:37.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd335b9a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c5590 0x7fd3301549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:37.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd335b9a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd330159000 0x7fd330154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:37.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd335b9a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd3300c5a90 con 0x7fd3300b2ec0 2024-03-28T15:08:37.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd335b9a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd33015f830 con 0x7fd3300c5590 2024-03-28T15:08:37.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd335b9a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd3301628e0 con 0x7fd330159000 2024-03-28T15:08:37.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd32f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3300b2ec0 0x7fd3301579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:37.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd32f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3300b2ec0 0x7fd3301579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43702/0 (socket says 172.21.15.186:43702) 2024-03-28T15:08:37.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd32f7fe640 1 -- 172.21.15.186:0/711813447 learned_addr learned my addr 172.21.15.186:0/711813447 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:37.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd32effd640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c5590 0x7fd3301549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd32ffff640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd330159000 0x7fd330154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd32f7fe640 1 -- 172.21.15.186:0/711813447 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd330159000 msgr2=0x7fd330154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd32f7fe640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd330159000 0x7fd330154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd32f7fe640 1 -- 172.21.15.186:0/711813447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c5590 msgr2=0x7fd3301549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd32f7fe640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c5590 0x7fd3301549d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.432+0000 7fd32f7fe640 1 -- 172.21.15.186:0/711813447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3300cfc50 con 0x7fd3300b2ec0 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.433+0000 7fd32f7fe640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3300b2ec0 0x7fd3301579c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd32000e990 tx=0x7fd32000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.433+0000 7fd317fff640 1 -- 172.21.15.186:0/711813447 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd320017070 con 0x7fd3300b2ec0 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.433+0000 7fd335b9a640 1 -- 172.21.15.186:0/711813447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3300c6ba0 con 0x7fd3300b2ec0 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.433+0000 7fd335b9a640 1 -- 172.21.15.186:0/711813447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd330158510 con 0x7fd3300b2ec0 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.433+0000 7fd317fff640 1 -- 172.21.15.186:0/711813447 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd320009400 con 0x7fd3300b2ec0 2024-03-28T15:08:37.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.433+0000 7fd317fff640 1 -- 172.21.15.186:0/711813447 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd320021060 con 0x7fd3300b2ec0 2024-03-28T15:08:37.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.435+0000 7fd335b9a640 1 -- 172.21.15.186:0/711813447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd310005290 con 0x7fd3300b2ec0 2024-03-28T15:08:37.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.435+0000 7fd317fff640 1 -- 172.21.15.186:0/711813447 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd32001a070 con 0x7fd3300b2ec0 2024-03-28T15:08:37.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.436+0000 7fd317fff640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd31c06c530 0x7fd31c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:37.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.436+0000 7fd317fff640 1 -- 172.21.15.186:0/711813447 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7fd32008b5b0 con 0x7fd3300b2ec0 2024-03-28T15:08:37.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.436+0000 7fd32effd640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd31c06c530 0x7fd31c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:37.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.436+0000 7fd32effd640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd31c06c530 0x7fd31c06e9f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd324036000 tx=0x7fd32403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:37.445 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.445+0000 7fd317fff640 1 -- 172.21.15.186:0/711813447 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd320059440 con 0x7fd3300b2ec0 2024-03-28T15:08:37.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.778+0000 7fd335b9a640 1 -- 172.21.15.186:0/711813447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd310005850 con 0x7fd3300b2ec0 2024-03-28T15:08:37.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.778+0000 7fd317fff640 1 -- 172.21.15.186:0/711813447 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6012 (secure 0 0 0) 0x7fd32005d740 con 0x7fd3300b2ec0 2024-03-28T15:08:37.779 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:37.779 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:31.221047+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:08:37.779 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 25 2024-03-28T15:08:37.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 -- 172.21.15.186:0/711813447 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd31c06c530 msgr2=0x7fd31c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:37.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd31c06c530 0x7fd31c06e9f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd324036000 tx=0x7fd32403b040 comp rx=0 tx=0).stop 2024-03-28T15:08:37.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 -- 172.21.15.186:0/711813447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3300b2ec0 msgr2=0x7fd3301579c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:37.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3300b2ec0 0x7fd3301579c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd32000e990 tx=0x7fd32000ee50 comp rx=0 tx=0).stop 2024-03-28T15:08:37.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 -- 172.21.15.186:0/711813447 shutdown_connections 2024-03-28T15:08:37.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd31c06c530 0x7fd31c06e9f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:37.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd330159000 0x7fd330154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:37.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c5590 0x7fd3301549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:37.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 --2- 172.21.15.186:0/711813447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3300b2ec0 0x7fd3301579c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:37.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 -- 172.21.15.186:0/711813447 >> 172.21.15.186:0/711813447 conn(0x7fd3300731f0 msgr2=0x7fd3300735d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:37.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 -- 172.21.15.186:0/711813447 shutdown_connections 2024-03-28T15:08:37.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:37.781+0000 7fd335b9a640 1 -- 172.21.15.186:0/711813447 wait complete. 2024-03-28T15:08:37.796 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:08:39.798 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:39.916 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.916+0000 7f1d59015640 1 Processor -- start 2024-03-28T15:08:39.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.916+0000 7f1d59015640 1 -- start start 2024-03-28T15:08:39.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.917+0000 7f1d59015640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1d540c82d0 0x7f1d540c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:39.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.917+0000 7f1d59015640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1d54067280 0x7f1d540676d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:39.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.917+0000 7f1d59015640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d540c94d0 con 0x7f1d54152ad0 2024-03-28T15:08:39.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.917+0000 7f1d59015640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f1d540c8fd0 con 0x7f1d540c82d0 2024-03-28T15:08:39.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.917+0000 7f1d5259c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1d540c82d0 0x7f1d540c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:39.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.917+0000 7f1d5259c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1d540c82d0 0x7f1d540c8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34824/0 (socket says 172.21.15.186:34824) 2024-03-28T15:08:39.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.917+0000 7f1d5259c640 1 -- 172.21.15.186:0/1809014453 learned_addr learned my addr 172.21.15.186:0/1809014453 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:39.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.917+0000 7f1d59015640 1 -- 172.21.15.186:0/1809014453 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f1d5406d850 con 0x7f1d54067280 2024-03-28T15:08:39.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.917+0000 7f1d5359e640 1 --2- 172.21.15.186:0/1809014453 >> v2:172.21.15.191:3300/0 conn(0x7f1d54067280 0x7f1d540676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:39.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.918+0000 7f1d5259c640 1 -- 172.21.15.186:0/1809014453 >> v2:172.21.15.191:3300/0 conn(0x7f1d54067280 msgr2=0x7f1d540676d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:39.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.918+0000 7f1d5259c640 1 --2- 172.21.15.186:0/1809014453 >> v2:172.21.15.191:3300/0 conn(0x7f1d54067280 0x7f1d540676d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:39.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.918+0000 7f1d51d9b640 1 -- 172.21.15.186:0/1809014453 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1551571303 0 0) 0x7f1d540c94d0 con 0x7f1d54152ad0 2024-03-28T15:08:39.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.918+0000 7f1d5259c640 1 -- 172.21.15.186:0/1809014453 >> v1:172.21.15.186:6789/0 conn(0x7f1d54152ad0 legacy=0x7f1d540c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:39.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.918+0000 7f1d5259c640 1 -- 172.21.15.186:0/1809014453 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d540c91f0 con 0x7f1d540c82d0 2024-03-28T15:08:39.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.918+0000 7f1d5259c640 1 --2- 172.21.15.186:0/1809014453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d540c82d0 0x7f1d540c8a90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1d540cbc30 tx=0x7f1d4802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e626a9ce53a980 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:39.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.919+0000 7f1d51d9b640 1 -- 172.21.15.186:0/1809014453 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d48035020 con 0x7f1d540c82d0 2024-03-28T15:08:39.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.919+0000 7f1d51d9b640 1 -- 172.21.15.186:0/1809014453 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d48031870 con 0x7f1d540c82d0 2024-03-28T15:08:39.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.919+0000 7f1d51d9b640 1 -- 172.21.15.186:0/1809014453 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d48031c40 con 0x7f1d540c82d0 2024-03-28T15:08:39.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.919+0000 7f1d59015640 1 -- 172.21.15.186:0/1809014453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d540c82d0 msgr2=0x7f1d540c8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:39.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.919+0000 7f1d59015640 1 --2- 172.21.15.186:0/1809014453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d540c82d0 0x7f1d540c8a90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1d540cbc30 tx=0x7f1d4802b910 comp rx=0 tx=0).stop 2024-03-28T15:08:39.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.919+0000 7f1d59015640 1 -- 172.21.15.186:0/1809014453 shutdown_connections 2024-03-28T15:08:39.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.919+0000 7f1d59015640 1 --2- 172.21.15.186:0/1809014453 >> v2:172.21.15.191:3300/0 conn(0x7f1d54067280 0x7f1d540676d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:39.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.919+0000 7f1d59015640 1 --2- 172.21.15.186:0/1809014453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d540c82d0 0x7f1d540c8a90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:39.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.919+0000 7f1d59015640 1 -- 172.21.15.186:0/1809014453 >> 172.21.15.186:0/1809014453 conn(0x7f1d5406fc80 msgr2=0x7f1d540734c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:39.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.920+0000 7f1d59015640 1 -- 172.21.15.186:0/1809014453 shutdown_connections 2024-03-28T15:08:39.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.920+0000 7f1d59015640 1 -- 172.21.15.186:0/1809014453 wait complete. 2024-03-28T15:08:39.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.922+0000 7f1d59015640 1 Processor -- start 2024-03-28T15:08:39.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.922+0000 7f1d59015640 1 -- start start 2024-03-28T15:08:39.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.922+0000 7f1d59015640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d54067280 0x7f1d540c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:39.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.922+0000 7f1d59015640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d540c82d0 0x7f1d540c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:39.925 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d59015640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d54152ad0 0x7f1d541eb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:39.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d59015640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1d540c4c20 con 0x7f1d54152ad0 2024-03-28T15:08:39.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d59015640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5406ddf0 con 0x7f1d540c82d0 2024-03-28T15:08:39.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d59015640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f1d540c4da0 con 0x7f1d54067280 2024-03-28T15:08:39.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d52d9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d54067280 0x7f1d540c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:39.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d5259c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d540c82d0 0x7f1d540c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:39.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d52d9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d54067280 0x7f1d540c5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:56266/0 (socket says 172.21.15.186:56266) 2024-03-28T15:08:39.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d52d9d640 1 -- 172.21.15.186:0/4220223548 learned_addr learned my addr 172.21.15.186:0/4220223548 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:39.926 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d5359e640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d54152ad0 0x7f1d541eb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d52d9d640 1 -- 172.21.15.186:0/4220223548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d540c82d0 msgr2=0x7f1d540c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d52d9d640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d540c82d0 0x7f1d540c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d52d9d640 1 -- 172.21.15.186:0/4220223548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d54152ad0 msgr2=0x7f1d541eb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d52d9d640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d54152ad0 0x7f1d541eb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d52d9d640 1 -- 172.21.15.186:0/4220223548 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d540c5530 con 0x7f1d54067280 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d5259c640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d540c82d0 0x7f1d540c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.923+0000 7f1d52d9d640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d54067280 0x7f1d540c5cf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400c490 tx=0x7f1d4400c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.924+0000 7f1d3b7fe640 1 -- 172.21.15.186:0/4220223548 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d44005480 con 0x7f1d54067280 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.924+0000 7f1d3b7fe640 1 -- 172.21.15.186:0/4220223548 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d440035f0 con 0x7f1d54067280 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.924+0000 7f1d3b7fe640 1 -- 172.21.15.186:0/4220223548 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d440039c0 con 0x7f1d54067280 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.924+0000 7f1d59015640 1 -- 172.21.15.186:0/4220223548 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d540c6d10 con 0x7f1d54067280 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.924+0000 7f1d59015640 1 -- 172.21.15.186:0/4220223548 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d540c57d0 con 0x7f1d54067280 2024-03-28T15:08:39.927 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.925+0000 7f1d59015640 1 -- 172.21.15.186:0/4220223548 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d54069500 con 0x7f1d54067280 2024-03-28T15:08:39.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.925+0000 7f1d3b7fe640 1 -- 172.21.15.186:0/4220223548 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1d44003b60 con 0x7f1d54067280 2024-03-28T15:08:39.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.926+0000 7f1d3b7fe640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1d4006c6d0 0x7f1d4006eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:39.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.926+0000 7f1d3b7fe640 1 -- 172.21.15.186:0/4220223548 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7f1d4408b3f0 con 0x7f1d54067280 2024-03-28T15:08:39.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.926+0000 7f1d5259c640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1d4006c6d0 0x7f1d4006eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:39.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.927+0000 7f1d5259c640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1d4006c6d0 0x7f1d4006eb90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1d48036000 tx=0x7f1d4803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:39.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:39.930+0000 7f1d3b7fe640 1 -- 172.21.15.186:0/4220223548 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f1d440591f0 con 0x7f1d54067280 2024-03-28T15:08:40.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.277+0000 7f1d59015640 1 -- 172.21.15.186:0/4220223548 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d540c7ea0 con 0x7f1d54067280 2024-03-28T15:08:40.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.278+0000 7f1d3b7fe640 1 -- 172.21.15.186:0/4220223548 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6016 (secure 0 0 0) 0x7f1d4405d4f0 con 0x7f1d54067280 2024-03-28T15:08:40.278 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:40.278 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:38.362728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:reconnect","state_seq":34,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:08:40.279 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 26 2024-03-28T15:08:40.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 -- 172.21.15.186:0/4220223548 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1d4006c6d0 msgr2=0x7f1d4006eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:40.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1d4006c6d0 0x7f1d4006eb90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1d48036000 tx=0x7f1d4803c040 comp rx=0 tx=0).stop 2024-03-28T15:08:40.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 -- 172.21.15.186:0/4220223548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d54067280 msgr2=0x7f1d540c5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:40.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d54067280 0x7f1d540c5cf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400c490 tx=0x7f1d4400c950 comp rx=0 tx=0).stop 2024-03-28T15:08:40.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 -- 172.21.15.186:0/4220223548 shutdown_connections 2024-03-28T15:08:40.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f1d4006c6d0 0x7f1d4006eb90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:40.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d54152ad0 0x7f1d541eb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:40.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d540c82d0 0x7f1d540c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:40.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 --2- 172.21.15.186:0/4220223548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d54067280 0x7f1d540c5cf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:40.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 -- 172.21.15.186:0/4220223548 >> 172.21.15.186:0/4220223548 conn(0x7f1d5406fc80 msgr2=0x7f1d54157180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:40.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 -- 172.21.15.186:0/4220223548 shutdown_connections 2024-03-28T15:08:40.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:40.280+0000 7f1d59015640 1 -- 172.21.15.186:0/4220223548 wait complete. 2024-03-28T15:08:40.293 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:08:42.294 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:42.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.420+0000 7f768c96c640 1 Processor -- start 2024-03-28T15:08:42.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.420+0000 7f768c96c640 1 -- start start 2024-03-28T15:08:42.422 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.421+0000 7f768c96c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7688152ad0 0x7f76880c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:42.422 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.421+0000 7f768c96c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f76880c82d0 0x7f76880c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:42.422 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.421+0000 7f768c96c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7688067280 0x7f7688067730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:42.422 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.421+0000 7f768c96c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f76880c8fd0 con 0x7f76880c82d0 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.421+0000 7f768c96c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f76880c35f0 con 0x7f7688067280 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.421+0000 7f768c96c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f76880c94d0 con 0x7f7688152ad0 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.421+0000 7f767ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f76880c82d0 0x7f76880c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.421+0000 7f767ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f76880c82d0 0x7f76880c30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34848/0 (socket says 172.21.15.186:34848) 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.421+0000 7f767ffff640 1 -- 172.21.15.186:0/783333660 learned_addr learned my addr 172.21.15.186:0/783333660 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.421+0000 7f768659c640 1 --2- 172.21.15.186:0/783333660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7688152ad0 0x7f76880c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.421+0000 7f7686d9d640 1 --2- 172.21.15.186:0/783333660 >> v2:172.21.15.191:3300/0 conn(0x7f7688067280 0x7f7688067730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.422+0000 7f767ffff640 1 -- 172.21.15.186:0/783333660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7688152ad0 msgr2=0x7f76880c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.422+0000 7f767ffff640 1 --2- 172.21.15.186:0/783333660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7688152ad0 0x7f76880c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.422+0000 7f767ffff640 1 -- 172.21.15.186:0/783333660 >> v2:172.21.15.191:3300/0 conn(0x7f7688067280 msgr2=0x7f7688067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.422+0000 7f767ffff640 1 --2- 172.21.15.186:0/783333660 >> v2:172.21.15.191:3300/0 conn(0x7f7688067280 0x7f7688067730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:42.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.422+0000 7f767ffff640 1 -- 172.21.15.186:0/783333660 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768806d850 con 0x7f76880c82d0 2024-03-28T15:08:42.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.422+0000 7f767ffff640 1 --2- 172.21.15.186:0/783333660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880c82d0 0x7f76880c30b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f76880c7900 tx=0x7f767002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8126902c111ffef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:42.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f7685d9b640 1 -- 172.21.15.186:0/783333660 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767002e070 con 0x7f76880c82d0 2024-03-28T15:08:42.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f7685d9b640 1 -- 172.21.15.186:0/783333660 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7670037070 con 0x7f76880c82d0 2024-03-28T15:08:42.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f7685d9b640 1 -- 172.21.15.186:0/783333660 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7670031a00 con 0x7f76880c82d0 2024-03-28T15:08:42.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f768c96c640 1 -- 172.21.15.186:0/783333660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880c82d0 msgr2=0x7f76880c30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:42.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f768c96c640 1 --2- 172.21.15.186:0/783333660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880c82d0 0x7f76880c30b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f76880c7900 tx=0x7f767002b880 comp rx=0 tx=0).stop 2024-03-28T15:08:42.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f768c96c640 1 -- 172.21.15.186:0/783333660 shutdown_connections 2024-03-28T15:08:42.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f768c96c640 1 --2- 172.21.15.186:0/783333660 >> v2:172.21.15.191:3300/0 conn(0x7f7688067280 0x7f7688067730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:42.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f768c96c640 1 --2- 172.21.15.186:0/783333660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880c82d0 0x7f76880c30b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:42.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f768c96c640 1 --2- 172.21.15.186:0/783333660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7688152ad0 0x7f76880c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:42.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f768c96c640 1 -- 172.21.15.186:0/783333660 >> 172.21.15.186:0/783333660 conn(0x7f768806fc80 msgr2=0x7f76880734c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:42.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f768c96c640 1 -- 172.21.15.186:0/783333660 shutdown_connections 2024-03-28T15:08:42.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.423+0000 7f768c96c640 1 -- 172.21.15.186:0/783333660 wait complete. 2024-03-28T15:08:42.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.425+0000 7f768c96c640 1 Processor -- start 2024-03-28T15:08:42.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.425+0000 7f768c96c640 1 -- start start 2024-03-28T15:08:42.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.425+0000 7f768c96c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688067280 0x7f76881f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:42.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.425+0000 7f768c96c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880c82d0 0x7f7688150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:42.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.425+0000 7f768c96c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7688152ad0 0x7f76881547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:42.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.425+0000 7f768c96c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f768806ddf0 con 0x7f76880c82d0 2024-03-28T15:08:42.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f768c96c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f76880c9820 con 0x7f7688067280 2024-03-28T15:08:42.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f768659c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688067280 0x7f76881f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:42.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f768659c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688067280 0x7f76881f21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:53762/0 (socket says 172.21.15.186:53762) 2024-03-28T15:08:42.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f768659c640 1 -- 172.21.15.186:0/2135589578 learned_addr learned my addr 172.21.15.186:0/2135589578 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f768c96c640 1 -- 172.21.15.186:0/2135589578 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f76880c4c20 con 0x7f7688152ad0 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f7686d9d640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7688152ad0 0x7f76881547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f767ffff640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880c82d0 0x7f7688150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f7686d9d640 1 -- 172.21.15.186:0/2135589578 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688067280 msgr2=0x7f76881f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f7686d9d640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688067280 0x7f76881f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f7686d9d640 1 -- 172.21.15.186:0/2135589578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880c82d0 msgr2=0x7f7688150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f7686d9d640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880c82d0 0x7f7688150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f7686d9d640 1 -- 172.21.15.186:0/2135589578 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76881f2730 con 0x7f7688152ad0 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f768659c640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688067280 0x7f76881f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f7686d9d640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7688152ad0 0x7f76881547a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f768000eab0 tx=0x7f768000ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f767effd640 1 -- 172.21.15.186:0/2135589578 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680018020 con 0x7f7688152ad0 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f767effd640 1 -- 172.21.15.186:0/2135589578 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f768000bc00 con 0x7f7688152ad0 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.426+0000 7f768c96c640 1 -- 172.21.15.186:0/2135589578 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7688158d30 con 0x7f7688152ad0 2024-03-28T15:08:42.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.427+0000 7f767effd640 1 -- 172.21.15.186:0/2135589578 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680014400 con 0x7f7688152ad0 2024-03-28T15:08:42.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.427+0000 7f768c96c640 1 -- 172.21.15.186:0/2135589578 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f76881592f0 con 0x7f7688152ad0 2024-03-28T15:08:42.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.428+0000 7f768c96c640 1 -- 172.21.15.186:0/2135589578 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7688069500 con 0x7f7688152ad0 2024-03-28T15:08:42.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.428+0000 7f767effd640 1 -- 172.21.15.186:0/2135589578 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7680005d10 con 0x7f7688152ad0 2024-03-28T15:08:42.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.428+0000 7f767effd640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f766806c530 0x7f766806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:42.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.429+0000 7f768659c640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f766806c530 0x7f766806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:42.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.429+0000 7f767effd640 1 -- 172.21.15.186:0/2135589578 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7f768008bc90 con 0x7f7688152ad0 2024-03-28T15:08:42.431 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.429+0000 7f768659c640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f766806c530 0x7f766806e9f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f76780112c0 tx=0x7f7678009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:42.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.433+0000 7f767effd640 1 -- 172.21.15.186:0/2135589578 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f7680022070 con 0x7f7688152ad0 2024-03-28T15:08:42.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.768+0000 7f768c96c640 1 -- 172.21.15.186:0/2135589578 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76880ca7a0 con 0x7f7688152ad0 2024-03-28T15:08:42.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.769+0000 7f767effd640 1 -- 172.21.15.186:0/2135589578 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6016 (secure 0 0 0) 0x7f7680059a90 con 0x7f7688152ad0 2024-03-28T15:08:42.769 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:42.769 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:38.362728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:reconnect","state_seq":34,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:08:42.770 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 26 2024-03-28T15:08:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 -- 172.21.15.186:0/2135589578 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f766806c530 msgr2=0x7f766806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f766806c530 0x7f766806e9f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f76780112c0 tx=0x7f7678009190 comp rx=0 tx=0).stop 2024-03-28T15:08:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 -- 172.21.15.186:0/2135589578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7688152ad0 msgr2=0x7f76881547a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7688152ad0 0x7f76881547a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f768000eab0 tx=0x7f768000ef70 comp rx=0 tx=0).stop 2024-03-28T15:08:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 -- 172.21.15.186:0/2135589578 shutdown_connections 2024-03-28T15:08:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f766806c530 0x7f766806e9f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7688152ad0 0x7f76881547a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76880c82d0 0x7f7688150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 --2- 172.21.15.186:0/2135589578 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688067280 0x7f76881f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 -- 172.21.15.186:0/2135589578 >> 172.21.15.186:0/2135589578 conn(0x7f768806fc80 msgr2=0x7f7688067660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 -- 172.21.15.186:0/2135589578 shutdown_connections 2024-03-28T15:08:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:42.772+0000 7f768c96c640 1 -- 172.21.15.186:0/2135589578 wait complete. 2024-03-28T15:08:42.785 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:08:44.786 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:44.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.901+0000 7f6b23684640 1 Processor -- start 2024-03-28T15:08:44.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.901+0000 7f6b23684640 1 -- start start 2024-03-28T15:08:44.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b23684640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6b1c151990 0x7f6b1c0c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:44.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b23684640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6b1c0b1e30 0x7f6b1c0c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:44.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b23684640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b1c06caa0 con 0x7f6b1c0c2d00 2024-03-28T15:08:44.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b23684640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6b1c0c1da0 con 0x7f6b1c151990 2024-03-28T15:08:44.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b23684640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f6b1c06cd80 con 0x7f6b1c0b1e30 2024-03-28T15:08:44.904 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b21420640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6b1c151990 0x7f6b1c0c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b21420640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6b1c151990 0x7f6b1c0c2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34852/0 (socket says 172.21.15.186:34852) 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b21420640 1 -- 172.21.15.186:0/1909614507 learned_addr learned my addr 172.21.15.186:0/1909614507 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b21c21640 1 --1- 172.21.15.186:0/1909614507 >> v1:172.21.15.186:6789/0 conn(0x7f6b1c0c2d00 0x7f6b1c0712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:49266/0 (socket says 172.21.15.186:49266) 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b20c1f640 1 --2- 172.21.15.186:0/1909614507 >> v2:172.21.15.191:3300/0 conn(0x7f6b1c0b1e30 0x7f6b1c0c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b21420640 1 -- 172.21.15.186:0/1909614507 >> v2:172.21.15.191:3300/0 conn(0x7f6b1c0b1e30 msgr2=0x7f6b1c0c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b21420640 1 --2- 172.21.15.186:0/1909614507 >> v2:172.21.15.191:3300/0 conn(0x7f6b1c0b1e30 0x7f6b1c0c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b13fff640 1 -- 172.21.15.186:0/1909614507 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3982356645 0 0) 0x7f6b1c06caa0 con 0x7f6b1c0c2d00 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b21420640 1 -- 172.21.15.186:0/1909614507 >> v1:172.21.15.186:6789/0 conn(0x7f6b1c0c2d00 legacy=0x7f6b1c0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.902+0000 7f6b21420640 1 -- 172.21.15.186:0/1909614507 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1c0c1340 con 0x7f6b1c151990 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.903+0000 7f6b21420640 1 --2- 172.21.15.186:0/1909614507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1c151990 0x7f6b1c0c2230 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c0724b0 tx=0x7f6b1802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b490ca7949797a07 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.903+0000 7f6b13fff640 1 -- 172.21.15.186:0/1909614507 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1802f850 con 0x7f6b1c151990 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.903+0000 7f6b13fff640 1 -- 172.21.15.186:0/1909614507 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b1802f9f0 con 0x7f6b1c151990 2024-03-28T15:08:44.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.903+0000 7f6b13fff640 1 -- 172.21.15.186:0/1909614507 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1802f850 con 0x7f6b1c151990 2024-03-28T15:08:44.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.904+0000 7f6b23684640 1 -- 172.21.15.186:0/1909614507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1c151990 msgr2=0x7f6b1c0c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:44.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.904+0000 7f6b23684640 1 --2- 172.21.15.186:0/1909614507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1c151990 0x7f6b1c0c2230 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c0724b0 tx=0x7f6b1802ba30 comp rx=0 tx=0).stop 2024-03-28T15:08:44.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.904+0000 7f6b23684640 1 -- 172.21.15.186:0/1909614507 shutdown_connections 2024-03-28T15:08:44.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.904+0000 7f6b23684640 1 --2- 172.21.15.186:0/1909614507 >> v2:172.21.15.191:3300/0 conn(0x7f6b1c0b1e30 0x7f6b1c0c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:44.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.904+0000 7f6b23684640 1 --2- 172.21.15.186:0/1909614507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1c151990 0x7f6b1c0c2230 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:44.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.904+0000 7f6b23684640 1 -- 172.21.15.186:0/1909614507 >> 172.21.15.186:0/1909614507 conn(0x7f6b1c0c8a20 msgr2=0x7f6b1c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:44.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.904+0000 7f6b23684640 1 -- 172.21.15.186:0/1909614507 shutdown_connections 2024-03-28T15:08:44.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.904+0000 7f6b23684640 1 -- 172.21.15.186:0/1909614507 wait complete. 2024-03-28T15:08:44.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.906+0000 7f6b23684640 1 Processor -- start 2024-03-28T15:08:44.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.906+0000 7f6b23684640 1 -- start start 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.907+0000 7f6b23684640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b1c0b1e30 0x7f6b1c156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.907+0000 7f6b23684640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c0c2d00 0x7f6b1c156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.907+0000 7f6b23684640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1c155760 0x7f6b1c0c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.907+0000 7f6b23684640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c073dd0 con 0x7f6b1c155760 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.907+0000 7f6b23684640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c0c1900 con 0x7f6b1c0b1e30 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.907+0000 7f6b23684640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f6b1c073f50 con 0x7f6b1c0c2d00 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.907+0000 7f6b21c21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1c155760 0x7f6b1c0c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.907+0000 7f6b21c21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1c155760 0x7f6b1c0c5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34862/0 (socket says 172.21.15.186:34862) 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.907+0000 7f6b21c21640 1 -- 172.21.15.186:0/3212559743 learned_addr learned my addr 172.21.15.186:0/3212559743 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.907+0000 7f6b21420640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b1c0b1e30 0x7f6b1c156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b20c1f640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c0c2d00 0x7f6b1c156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:44.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b21c21640 1 -- 172.21.15.186:0/3212559743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c0c2d00 msgr2=0x7f6b1c156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:44.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b21c21640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c0c2d00 0x7f6b1c156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:44.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b21c21640 1 -- 172.21.15.186:0/3212559743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b1c0b1e30 msgr2=0x7f6b1c156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:44.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b21c21640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b1c0b1e30 0x7f6b1c156420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:44.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b21c21640 1 -- 172.21.15.186:0/3212559743 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1c158190 con 0x7f6b1c155760 2024-03-28T15:08:44.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b20c1f640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c0c2d00 0x7f6b1c156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:08:44.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b21c21640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1c155760 0x7f6b1c0c5980 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6b14001de0 tx=0x7f6b14010420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:44.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b11ffb640 1 -- 172.21.15.186:0/3212559743 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1400e110 con 0x7f6b1c155760 2024-03-28T15:08:44.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b11ffb640 1 -- 172.21.15.186:0/3212559743 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b1400e2b0 con 0x7f6b1c155760 2024-03-28T15:08:44.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b23684640 1 -- 172.21.15.186:0/3212559743 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b1c1541e0 con 0x7f6b1c155760 2024-03-28T15:08:44.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.908+0000 7f6b11ffb640 1 -- 172.21.15.186:0/3212559743 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b14004d80 con 0x7f6b1c155760 2024-03-28T15:08:44.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.909+0000 7f6b23684640 1 -- 172.21.15.186:0/3212559743 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b1c1547a0 con 0x7f6b1c155760 2024-03-28T15:08:44.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.910+0000 7f6b23684640 1 -- 172.21.15.186:0/3212559743 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b1c140080 con 0x7f6b1c155760 2024-03-28T15:08:44.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.910+0000 7f6b11ffb640 1 -- 172.21.15.186:0/3212559743 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6b14010b30 con 0x7f6b1c155760 2024-03-28T15:08:44.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.911+0000 7f6b11ffb640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6b0006c530 0x7f6b0006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:44.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.911+0000 7f6b21420640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6b0006c530 0x7f6b0006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:44.912 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.912+0000 7f6b21420640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6b0006c530 0x7f6b0006e9f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6b18036000 tx=0x7f6b18008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:44.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.912+0000 7f6b11ffb640 1 -- 172.21.15.186:0/3212559743 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7f6b1405ac20 con 0x7f6b1c155760 2024-03-28T15:08:44.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:44.920+0000 7f6b11ffb640 1 -- 172.21.15.186:0/3212559743 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6b14010e20 con 0x7f6b1c155760 2024-03-28T15:08:45.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.240+0000 7f6b23684640 1 -- 172.21.15.186:0/3212559743 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b1c069500 con 0x7f6b1c155760 2024-03-28T15:08:45.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.241+0000 7f6b11ffb640 1 -- 172.21.15.186:0/3212559743 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6013 (secure 0 0 0) 0x7f6b1405d280 con 0x7f6b1c155760 2024-03-28T15:08:45.242 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:45.242 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:43.831405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:rejoin","state_seq":36,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:08:45.242 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 27 2024-03-28T15:08:45.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 -- 172.21.15.186:0/3212559743 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6b0006c530 msgr2=0x7f6b0006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:45.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6b0006c530 0x7f6b0006e9f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6b18036000 tx=0x7f6b18008030 comp rx=0 tx=0).stop 2024-03-28T15:08:45.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 -- 172.21.15.186:0/3212559743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1c155760 msgr2=0x7f6b1c0c5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:45.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1c155760 0x7f6b1c0c5980 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6b14001de0 tx=0x7f6b14010420 comp rx=0 tx=0).stop 2024-03-28T15:08:45.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 -- 172.21.15.186:0/3212559743 shutdown_connections 2024-03-28T15:08:45.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6b0006c530 0x7f6b0006e9f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:45.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1c155760 0x7f6b1c0c5980 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:45.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c0c2d00 0x7f6b1c156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:45.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 --2- 172.21.15.186:0/3212559743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b1c0b1e30 0x7f6b1c156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:45.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 -- 172.21.15.186:0/3212559743 >> 172.21.15.186:0/3212559743 conn(0x7f6b1c0c8a20 msgr2=0x7f6b1c14fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:45.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 -- 172.21.15.186:0/3212559743 shutdown_connections 2024-03-28T15:08:45.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:45.244+0000 7f6b23684640 1 -- 172.21.15.186:0/3212559743 wait complete. 2024-03-28T15:08:45.259 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T15:01:31.425299+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5137": { "addr": "172.21.15.191:6837/3037586083", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3037586083, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3037586083, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5137, "incarnation": 25, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:rejoin", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T15:08:43.831405+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5137 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/1278716016", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6833", "nonce": 1278716016, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1278716016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6835/2068761001", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2068761001, "type": "v2" }, { "addr": "172.21.15.186:6835", "nonce": 2068761001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2344737463", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6837", "nonce": 2344737463, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2344737463, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4284, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/1548903381", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 1548903381, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 1548903381, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T15:08:47.260 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:47.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.385+0000 7fc6cd75e640 1 Processor -- start 2024-03-28T15:08:47.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.385+0000 7fc6cd75e640 1 -- start start 2024-03-28T15:08:47.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.385+0000 7fc6cd75e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6c80c53c0 0x7fc6c80c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:47.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.385+0000 7fc6cd75e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc6c80c57e0 0x7fc6c80cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.385+0000 7fc6cd75e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6c815b080 con 0x7fc6c8154bd0 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.385+0000 7fc6cd75e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc6c80c5ca0 con 0x7fc6c80c53c0 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.385+0000 7fc6cd75e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fc6c815b360 con 0x7fc6c80c57e0 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.386+0000 7fc6c67fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6c80c53c0 0x7fc6c80c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.386+0000 7fc6c67fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6c80c53c0 0x7fc6c80c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34868/0 (socket says 172.21.15.186:34868) 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.386+0000 7fc6c67fc640 1 -- 172.21.15.186:0/4215437957 learned_addr learned my addr 172.21.15.186:0/4215437957 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.386+0000 7fc6c77fe640 1 --2- 172.21.15.186:0/4215437957 >> v2:172.21.15.191:3300/0 conn(0x7fc6c80c57e0 0x7fc6c80cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.386+0000 7fc6c67fc640 1 -- 172.21.15.186:0/4215437957 >> v2:172.21.15.191:3300/0 conn(0x7fc6c80c57e0 msgr2=0x7fc6c80cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.386+0000 7fc6c67fc640 1 --2- 172.21.15.186:0/4215437957 >> v2:172.21.15.191:3300/0 conn(0x7fc6c80c57e0 0x7fc6c80cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.386+0000 7fc6c67fc640 1 -- 172.21.15.186:0/4215437957 >> v1:172.21.15.186:6789/0 conn(0x7fc6c8154bd0 legacy=0x7fc6c815a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.386+0000 7fc6c67fc640 1 -- 172.21.15.186:0/4215437957 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c815b640 con 0x7fc6c80c53c0 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.387+0000 7fc6c67fc640 1 --2- 172.21.15.186:0/4215437957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6c80c53c0 0x7fc6c80c66f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80b4180 tx=0x7fc6bc02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52500ac2c0df2e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.387+0000 7fc6c5ffb640 1 -- 172.21.15.186:0/4215437957 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc02f850 con 0x7fc6c80c53c0 2024-03-28T15:08:47.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.387+0000 7fc6c5ffb640 1 -- 172.21.15.186:0/4215437957 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6bc02f9f0 con 0x7fc6c80c53c0 2024-03-28T15:08:47.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.387+0000 7fc6c5ffb640 1 -- 172.21.15.186:0/4215437957 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc02fdc0 con 0x7fc6c80c53c0 2024-03-28T15:08:47.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.388+0000 7fc6cd75e640 1 -- 172.21.15.186:0/4215437957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6c80c53c0 msgr2=0x7fc6c80c66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:47.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.388+0000 7fc6cd75e640 1 --2- 172.21.15.186:0/4215437957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6c80c53c0 0x7fc6c80c66f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80b4180 tx=0x7fc6bc02ba30 comp rx=0 tx=0).stop 2024-03-28T15:08:47.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.388+0000 7fc6cd75e640 1 -- 172.21.15.186:0/4215437957 shutdown_connections 2024-03-28T15:08:47.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.388+0000 7fc6cd75e640 1 --2- 172.21.15.186:0/4215437957 >> v2:172.21.15.191:3300/0 conn(0x7fc6c80c57e0 0x7fc6c80cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:47.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.388+0000 7fc6cd75e640 1 --2- 172.21.15.186:0/4215437957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6c80c53c0 0x7fc6c80c66f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:47.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.388+0000 7fc6cd75e640 1 -- 172.21.15.186:0/4215437957 >> 172.21.15.186:0/4215437957 conn(0x7fc6c80731f0 msgr2=0x7fc6c8073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:47.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.388+0000 7fc6cd75e640 1 -- 172.21.15.186:0/4215437957 shutdown_connections 2024-03-28T15:08:47.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.388+0000 7fc6cd75e640 1 -- 172.21.15.186:0/4215437957 wait complete. 2024-03-28T15:08:47.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.389+0000 7fc6cd75e640 1 Processor -- start 2024-03-28T15:08:47.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.389+0000 7fc6cd75e640 1 -- start start 2024-03-28T15:08:47.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.390+0000 7fc6cd75e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6c80c53c0 0x7fc6c81540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:47.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.390+0000 7fc6cd75e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6c80c57e0 0x7fc6c8143490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.390+0000 7fc6cd75e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6c8154bd0 0x7fc6c81439d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.390+0000 7fc6cd75e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c80cc390 con 0x7fc6c80c53c0 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.390+0000 7fc6c6ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6c80c53c0 0x7fc6c81540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.390+0000 7fc6cd75e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c80c61d0 con 0x7fc6c80c57e0 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.390+0000 7fc6c6ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6c80c53c0 0x7fc6c81540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34884/0 (socket says 172.21.15.186:34884) 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.390+0000 7fc6c6ffd640 1 -- 172.21.15.186:0/3374020410 learned_addr learned my addr 172.21.15.186:0/3374020410 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.390+0000 7fc6cd75e640 1 -- 172.21.15.186:0/3374020410 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc6c80c6c30 con 0x7fc6c8154bd0 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.390+0000 7fc6c67fc640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6c80c57e0 0x7fc6c8143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6c77fe640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6c8154bd0 0x7fc6c81439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6c6ffd640 1 -- 172.21.15.186:0/3374020410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6c8154bd0 msgr2=0x7fc6c81439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6c6ffd640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6c8154bd0 0x7fc6c81439d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6c6ffd640 1 -- 172.21.15.186:0/3374020410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6c80c57e0 msgr2=0x7fc6c8143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6c6ffd640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6c80c57e0 0x7fc6c8143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:47.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6c6ffd640 1 -- 172.21.15.186:0/3374020410 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c8153b30 con 0x7fc6c80c53c0 2024-03-28T15:08:47.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6c6ffd640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6c80c53c0 0x7fc6c81540d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800c400 tx=0x7fc6b800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:47.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6af7fe640 1 -- 172.21.15.186:0/3374020410 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b80052d0 con 0x7fc6c80c53c0 2024-03-28T15:08:47.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6af7fe640 1 -- 172.21.15.186:0/3374020410 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b8005490 con 0x7fc6c80c53c0 2024-03-28T15:08:47.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6cd75e640 1 -- 172.21.15.186:0/3374020410 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c80c8660 con 0x7fc6c80c53c0 2024-03-28T15:08:47.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6cd75e640 1 -- 172.21.15.186:0/3374020410 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c80c8c20 con 0x7fc6c80c53c0 2024-03-28T15:08:47.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.391+0000 7fc6af7fe640 1 -- 172.21.15.186:0/3374020410 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8007260 con 0x7fc6c80c53c0 2024-03-28T15:08:47.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.393+0000 7fc6cd75e640 1 -- 172.21.15.186:0/3374020410 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c80cf910 con 0x7fc6c80c53c0 2024-03-28T15:08:47.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.393+0000 7fc6af7fe640 1 -- 172.21.15.186:0/3374020410 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc6b8007400 con 0x7fc6c80c53c0 2024-03-28T15:08:47.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.393+0000 7fc6af7fe640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc6b406c530 0x7fc6b406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:47.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.394+0000 7fc6af7fe640 1 -- 172.21.15.186:0/3374020410 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7fc6b808ae10 con 0x7fc6c80c53c0 2024-03-28T15:08:47.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.394+0000 7fc6c67fc640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc6b406c530 0x7fc6b406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:47.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.394+0000 7fc6c67fc640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc6b406c530 0x7fc6b406e9f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc036000 tx=0x7fc6bc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:47.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.398+0000 7fc6af7fe640 1 -- 172.21.15.186:0/3374020410 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc6c80cf910 con 0x7fc6c80c53c0 2024-03-28T15:08:47.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.711+0000 7fc6cd75e640 1 -- 172.21.15.186:0/3374020410 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6c8069500 con 0x7fc6c80c53c0 2024-03-28T15:08:47.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.712+0000 7fc6af7fe640 1 -- 172.21.15.186:0/3374020410 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6013 (secure 0 0 0) 0x7fc6b8058b90 con 0x7fc6c80c53c0 2024-03-28T15:08:47.713 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:47.713 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:43.831405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:rejoin","state_seq":36,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:08:47.713 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 27 2024-03-28T15:08:47.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 -- 172.21.15.186:0/3374020410 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc6b406c530 msgr2=0x7fc6b406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:47.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc6b406c530 0x7fc6b406e9f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc036000 tx=0x7fc6bc008030 comp rx=0 tx=0).stop 2024-03-28T15:08:47.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 -- 172.21.15.186:0/3374020410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6c80c53c0 msgr2=0x7fc6c81540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:47.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6c80c53c0 0x7fc6c81540d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800c400 tx=0x7fc6b800c8c0 comp rx=0 tx=0).stop 2024-03-28T15:08:47.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 -- 172.21.15.186:0/3374020410 shutdown_connections 2024-03-28T15:08:47.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc6b406c530 0x7fc6b406e9f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:47.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc6c8154bd0 0x7fc6c81439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:47.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6c80c57e0 0x7fc6c8143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:47.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 --2- 172.21.15.186:0/3374020410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6c80c53c0 0x7fc6c81540d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:47.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 -- 172.21.15.186:0/3374020410 >> 172.21.15.186:0/3374020410 conn(0x7fc6c80731f0 msgr2=0x7fc6c80735d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:47.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 -- 172.21.15.186:0/3374020410 shutdown_connections 2024-03-28T15:08:47.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:47.714+0000 7fc6cd75e640 1 -- 172.21.15.186:0/3374020410 wait complete. 2024-03-28T15:08:47.726 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:08:49.726 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:49.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.848+0000 7f2303ac3640 1 Processor -- start 2024-03-28T15:08:49.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.848+0000 7f2303ac3640 1 -- start start 2024-03-28T15:08:49.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.849+0000 7f2303ac3640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f22fc15a780 0x7f22fc15ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.849+0000 7f2303ac3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22fc06caf0 con 0x7f22fc06c390 2024-03-28T15:08:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.849+0000 7f2303ac3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22fc06cd90 con 0x7f22fc154bf0 2024-03-28T15:08:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.849+0000 7f2303ac3640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f22fc15b0d0 con 0x7f22fc15a780 2024-03-28T15:08:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.849+0000 7f230105e640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f22fc06c390 0x7f22fc06e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:55054/0 (socket says 172.21.15.186:55054) 2024-03-28T15:08:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.849+0000 7f230105e640 1 -- 172.21.15.186:0/2577924845 learned_addr learned my addr 172.21.15.186:0/2577924845 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:08:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.850+0000 7f2302060640 1 --2- 172.21.15.186:0/2577924845 >> v2:172.21.15.191:3300/0 conn(0x7f22fc15a780 0x7f22fc15ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.850+0000 7f230085d640 1 -- 172.21.15.186:0/2577924845 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2633810454 0 0) 0x7f22fc06caf0 con 0x7f22fc06c390 2024-03-28T15:08:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.850+0000 7f230085d640 1 -- 172.21.15.186:0/2577924845 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22e8003520 con 0x7f22fc06c390 2024-03-28T15:08:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.850+0000 7f2302060640 1 -- 172.21.15.186:0/2577924845 >> v1:172.21.15.191:6789/0 conn(0x7f22fc154bf0 legacy=0x7f22fc072410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.850+0000 7f2302060640 1 -- 172.21.15.186:0/2577924845 >> v1:172.21.15.186:6789/0 conn(0x7f22fc06c390 legacy=0x7f22fc06e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.850+0000 7f2302060640 1 -- 172.21.15.186:0/2577924845 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22fc06b500 con 0x7f22fc15a780 2024-03-28T15:08:49.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.851+0000 7f2302060640 1 --2- 172.21.15.186:0/2577924845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22fc15a780 0x7f22fc15ab90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009d90 tx=0x7f22f402bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cc37fa7dad84ae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.852+0000 7f230085d640 1 -- 172.21.15.186:0/2577924845 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4039070 con 0x7f22fc15a780 2024-03-28T15:08:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.852+0000 7f230085d640 1 -- 172.21.15.186:0/2577924845 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f402cd80 con 0x7f22fc15a780 2024-03-28T15:08:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.852+0000 7f230085d640 1 -- 172.21.15.186:0/2577924845 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4034660 con 0x7f22fc15a780 2024-03-28T15:08:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.852+0000 7f2303ac3640 1 -- 172.21.15.186:0/2577924845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22fc15a780 msgr2=0x7f22fc15ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.852+0000 7f2303ac3640 1 --2- 172.21.15.186:0/2577924845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22fc15a780 0x7f22fc15ab90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009d90 tx=0x7f22f402bfd0 comp rx=0 tx=0).stop 2024-03-28T15:08:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.852+0000 7f2303ac3640 1 -- 172.21.15.186:0/2577924845 shutdown_connections 2024-03-28T15:08:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.852+0000 7f2303ac3640 1 --2- 172.21.15.186:0/2577924845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22fc15a780 0x7f22fc15ab90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.852+0000 7f2303ac3640 1 -- 172.21.15.186:0/2577924845 >> 172.21.15.186:0/2577924845 conn(0x7f22fc0cd360 msgr2=0x7f22fc0cd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.852+0000 7f2303ac3640 1 -- 172.21.15.186:0/2577924845 shutdown_connections 2024-03-28T15:08:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.853+0000 7f2303ac3640 1 -- 172.21.15.186:0/2577924845 wait complete. 2024-03-28T15:08:49.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.854+0000 7f2303ac3640 1 Processor -- start 2024-03-28T15:08:49.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.854+0000 7f2303ac3640 1 -- start start 2024-03-28T15:08:49.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.855+0000 7f2303ac3640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22fc06c390 0x7f22fc0c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:49.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.855+0000 7f2303ac3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22fc154bf0 0x7f22fc1530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:49.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.855+0000 7f2303ac3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22fc15a780 0x7f22fc1f1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:49.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.855+0000 7f2303ac3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc06bda0 con 0x7f22fc154bf0 2024-03-28T15:08:49.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.855+0000 7f2303ac3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc06bac0 con 0x7f22fc15a780 2024-03-28T15:08:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.855+0000 7f2303ac3640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f22fc06c7d0 con 0x7f22fc06c390 2024-03-28T15:08:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.855+0000 7f230105e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22fc154bf0 0x7f22fc1530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.855+0000 7f230105e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22fc154bf0 0x7f22fc1530b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37048/0 (socket says 172.21.15.186:37048) 2024-03-28T15:08:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.855+0000 7f230105e640 1 -- 172.21.15.186:0/3842165108 learned_addr learned my addr 172.21.15.186:0/3842165108 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.856+0000 7f230105e640 1 -- 172.21.15.186:0/3842165108 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22fc06c390 msgr2=0x7f22fc0c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.856+0000 7f230105e640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22fc06c390 0x7f22fc0c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.856+0000 7f230105e640 1 -- 172.21.15.186:0/3842165108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22fc15a780 msgr2=0x7f22fc1f1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.856+0000 7f230105e640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22fc15a780 0x7f22fc1f1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.856+0000 7f230105e640 1 -- 172.21.15.186:0/3842165108 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22fc0c8130 con 0x7f22fc154bf0 2024-03-28T15:08:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.856+0000 7f230105e640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22fc154bf0 0x7f22fc1530b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00c410 tx=0x7f22ec00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.859+0000 7f22f27fc640 1 -- 172.21.15.186:0/3842165108 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec017070 con 0x7f22fc154bf0 2024-03-28T15:08:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.859+0000 7f22f27fc640 1 -- 172.21.15.186:0/3842165108 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22ec005ce0 con 0x7f22fc154bf0 2024-03-28T15:08:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.859+0000 7f22f27fc640 1 -- 172.21.15.186:0/3842165108 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec0072d0 con 0x7f22fc154bf0 2024-03-28T15:08:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.859+0000 7f2303ac3640 1 -- 172.21.15.186:0/3842165108 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22fc0c86d0 con 0x7f22fc154bf0 2024-03-28T15:08:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.859+0000 7f2303ac3640 1 -- 172.21.15.186:0/3842165108 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22fc0c8ab0 con 0x7f22fc154bf0 2024-03-28T15:08:49.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.861+0000 7f22f27fc640 1 -- 172.21.15.186:0/3842165108 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f22ec01a070 con 0x7f22fc154bf0 2024-03-28T15:08:49.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.861+0000 7f22f27fc640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f22e006c530 0x7f22e006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:49.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.861+0000 7f22f27fc640 1 -- 172.21.15.186:0/3842165108 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7f22ec08ac30 con 0x7f22fc154bf0 2024-03-28T15:08:49.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.861+0000 7f230185f640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f22e006c530 0x7f22e006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:49.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.862+0000 7f230185f640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f22e006c530 0x7f22e006e9f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f22f8000f60 tx=0x7f22f80079d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:49.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.862+0000 7f2303ac3640 1 -- 172.21.15.186:0/3842165108 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22fc06fdc0 con 0x7f22fc154bf0 2024-03-28T15:08:49.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:49.871+0000 7f22f27fc640 1 -- 172.21.15.186:0/3842165108 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f22fc06fdc0 con 0x7f22fc154bf0 2024-03-28T15:08:50.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.258+0000 7f2303ac3640 1 -- 172.21.15.186:0/3842165108 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22fc0ccc10 con 0x7f22fc154bf0 2024-03-28T15:08:50.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.259+0000 7f22f27fc640 1 -- 172.21.15.186:0/3842165108 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6020 (secure 0 0 0) 0x7f22fc0ccc10 con 0x7f22fc154bf0 2024-03-28T15:08:50.259 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:50.259 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:49.855657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:active","state_seq":38,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5137,"qdb_cluster":[5137]},"id":1}]} 2024-03-28T15:08:50.260 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 28 2024-03-28T15:08:50.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.261+0000 7f2303ac3640 1 -- 172.21.15.186:0/3842165108 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f22e006c530 msgr2=0x7f22e006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:50.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.262+0000 7f2303ac3640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f22e006c530 0x7f22e006e9f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f22f8000f60 tx=0x7f22f80079d0 comp rx=0 tx=0).stop 2024-03-28T15:08:50.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.262+0000 7f2303ac3640 1 -- 172.21.15.186:0/3842165108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22fc154bf0 msgr2=0x7f22fc1530b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:50.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.262+0000 7f2303ac3640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22fc154bf0 0x7f22fc1530b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00c410 tx=0x7f22ec00c8d0 comp rx=0 tx=0).stop 2024-03-28T15:08:50.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.262+0000 7f2303ac3640 1 -- 172.21.15.186:0/3842165108 shutdown_connections 2024-03-28T15:08:50.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.262+0000 7f2303ac3640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f22e006c530 0x7f22e006e9f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:50.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.262+0000 7f2303ac3640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22fc15a780 0x7f22fc1f1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:50.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.262+0000 7f2303ac3640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22fc154bf0 0x7f22fc1530b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:50.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.262+0000 7f2303ac3640 1 --2- 172.21.15.186:0/3842165108 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22fc06c390 0x7f22fc0c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:50.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.262+0000 7f2303ac3640 1 -- 172.21.15.186:0/3842165108 >> 172.21.15.186:0/3842165108 conn(0x7f22fc0cd360 msgr2=0x7f22fc06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:50.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.262+0000 7f2303ac3640 1 -- 172.21.15.186:0/3842165108 shutdown_connections 2024-03-28T15:08:50.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:50.262+0000 7f2303ac3640 1 -- 172.21.15.186:0/3842165108 wait complete. 2024-03-28T15:08:50.276 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=4790 2024-03-28T15:08:50.276 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.b 2024-03-28T15:08:55.277 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-03-28T15:08:55.277 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T15:08:55.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T15:08:55.280 INFO:tasks.ceph.mds.b:Started 2024-03-28T15:08:55.280 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:55.358 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2024-03-28T15:08:55.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.389+0000 7f5b5088a640 1 Processor -- start 2024-03-28T15:08:55.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.389+0000 7f5b5088a640 1 -- start start 2024-03-28T15:08:55.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b5088a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5b4c154dc0 0x7f5b4c072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b5088a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b4c0cb7a0 con 0x7f5b4c0cb180 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b5088a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b4c15eb70 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b5088a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f5b4c071c20 con 0x7f5b4c154dc0 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b4a59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5b4c154dc0 0x7f5b4c072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b4a59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5b4c154dc0 0x7f5b4c072420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37052/0 (socket says 172.21.15.186:37052) 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b4a59c640 1 -- 172.21.15.186:0/165947915 learned_addr learned my addr 172.21.15.186:0/165947915 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b49d9b640 1 --1- 172.21.15.186:0/165947915 >> v1:172.21.15.191:6789/0 conn(0x7f5b4c0c52f0 0x7f5b4c0caa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:36278/0 (socket says 172.21.15.186:36278) 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b4ad9d640 1 --1- 172.21.15.186:0/165947915 >> v1:172.21.15.186:6789/0 conn(0x7f5b4c0cb180 0x7f5b4c165210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:55056/0 (socket says 172.21.15.186:55056) 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b4959a640 1 -- 172.21.15.186:0/165947915 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1284942405 0 0) 0x7f5b4c15eb70 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b4a59c640 1 -- 172.21.15.186:0/165947915 >> v1:172.21.15.191:6789/0 conn(0x7f5b4c0c52f0 legacy=0x7f5b4c0caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b4a59c640 1 -- 172.21.15.186:0/165947915 >> v1:172.21.15.186:6789/0 conn(0x7f5b4c0cb180 legacy=0x7f5b4c165210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:55.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.390+0000 7f5b4a59c640 1 -- 172.21.15.186:0/165947915 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4c071f00 con 0x7f5b4c154dc0 2024-03-28T15:08:55.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.391+0000 7f5b4a59c640 1 --2- 172.21.15.186:0/165947915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b4c154dc0 0x7f5b4c072420 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009870 tx=0x7f5b3c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f091ed1437d12e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:55.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.391+0000 7f5b4959a640 1 -- 172.21.15.186:0/165947915 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c038070 con 0x7f5b4c154dc0 2024-03-28T15:08:55.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.391+0000 7f5b4959a640 1 -- 172.21.15.186:0/165947915 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b3c031740 con 0x7f5b4c154dc0 2024-03-28T15:08:55.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.391+0000 7f5b4959a640 1 -- 172.21.15.186:0/165947915 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c02f050 con 0x7f5b4c154dc0 2024-03-28T15:08:55.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.392+0000 7f5b5088a640 1 -- 172.21.15.186:0/165947915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b4c154dc0 msgr2=0x7f5b4c072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:55.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.392+0000 7f5b5088a640 1 --2- 172.21.15.186:0/165947915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b4c154dc0 0x7f5b4c072420 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009870 tx=0x7f5b3c02b860 comp rx=0 tx=0).stop 2024-03-28T15:08:55.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.392+0000 7f5b5088a640 1 -- 172.21.15.186:0/165947915 shutdown_connections 2024-03-28T15:08:55.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.392+0000 7f5b5088a640 1 --2- 172.21.15.186:0/165947915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b4c154dc0 0x7f5b4c072420 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:55.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.392+0000 7f5b5088a640 1 -- 172.21.15.186:0/165947915 >> 172.21.15.186:0/165947915 conn(0x7f5b4c0d00f0 msgr2=0x7f5b4c0d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:55.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.392+0000 7f5b5088a640 1 -- 172.21.15.186:0/165947915 shutdown_connections 2024-03-28T15:08:55.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.392+0000 7f5b5088a640 1 -- 172.21.15.186:0/165947915 wait complete. 2024-03-28T15:08:55.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.394+0000 7f5b5088a640 1 Processor -- start 2024-03-28T15:08:55.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.394+0000 7f5b5088a640 1 -- start start 2024-03-28T15:08:55.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.394+0000 7f5b5088a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b4c0c52f0 0x7f5b4c151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:55.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b5088a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b4c0cb180 0x7f5b4c1e7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b5088a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b4c154dc0 0x7f5b4c150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b5088a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4c0c5840 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b5088a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4c15e840 con 0x7f5b4c0cb180 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b5088a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5b4c0c4fd0 con 0x7f5b4c154dc0 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b4a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b4c0c52f0 0x7f5b4c151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b4a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b4c0c52f0 0x7f5b4c151320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37056/0 (socket says 172.21.15.186:37056) 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b4a59c640 1 -- 172.21.15.186:0/531581786 learned_addr learned my addr 172.21.15.186:0/531581786 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b49d9b640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b4c0cb180 0x7f5b4c1e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b4ad9d640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b4c154dc0 0x7f5b4c150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b4a59c640 1 -- 172.21.15.186:0/531581786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b4c154dc0 msgr2=0x7f5b4c150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b4a59c640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b4c154dc0 0x7f5b4c150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b4a59c640 1 -- 172.21.15.186:0/531581786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b4c0cb180 msgr2=0x7f5b4c1e7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b4a59c640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b4c0cb180 0x7f5b4c1e7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b4a59c640 1 -- 172.21.15.186:0/531581786 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4c151860 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b4a59c640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b4c0c52f0 0x7f5b4c151320 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009870 tx=0x7f5b3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:55.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b3affd640 1 -- 172.21.15.186:0/531581786 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c042070 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b3affd640 1 -- 172.21.15.186:0/531581786 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b3c033a70 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b5088a640 1 -- 172.21.15.186:0/531581786 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b4c0cee60 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b3affd640 1 -- 172.21.15.186:0/531581786 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c038040 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.395+0000 7f5b5088a640 1 -- 172.21.15.186:0/531581786 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b4c0cfdc0 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.398 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.396+0000 7f5b5088a640 1 -- 172.21.15.186:0/531581786 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b4c069500 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.397+0000 7f5b3affd640 1 -- 172.21.15.186:0/531581786 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5b3c045070 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.397+0000 7f5b3affd640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5b2c06c530 0x7f5b2c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:55.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.398+0000 7f5b3affd640 1 -- 172.21.15.186:0/531581786 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7f5b3c0ad1e0 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.398+0000 7f5b49d9b640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5b2c06c530 0x7f5b2c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:55.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.398+0000 7f5b49d9b640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5b2c06c530 0x7f5b2c06e9f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5b40007f70 tx=0x7f5b40007f00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:55.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.403+0000 7f5b3affd640 1 -- 172.21.15.186:0/531581786 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5b3c04c070 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.675+0000 7f5b5088a640 1 -- 172.21.15.186:0/531581786 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b4c14ee40 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.675+0000 7f5b3affd640 1 -- 172.21.15.186:0/531581786 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6020 (secure 0 0 0) 0x7f5b3c07afd0 con 0x7f5b4c0c52f0 2024-03-28T15:08:55.676 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:55.676 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:49.855657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:active","state_seq":38,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5137,"qdb_cluster":[5137]},"id":1}]} 2024-03-28T15:08:55.676 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 28 2024-03-28T15:08:55.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 -- 172.21.15.186:0/531581786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5b2c06c530 msgr2=0x7f5b2c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:55.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5b2c06c530 0x7f5b2c06e9f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5b40007f70 tx=0x7f5b40007f00 comp rx=0 tx=0).stop 2024-03-28T15:08:55.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 -- 172.21.15.186:0/531581786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b4c0c52f0 msgr2=0x7f5b4c151320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:55.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b4c0c52f0 0x7f5b4c151320 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009870 tx=0x7f5b3c0046c0 comp rx=0 tx=0).stop 2024-03-28T15:08:55.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 -- 172.21.15.186:0/531581786 shutdown_connections 2024-03-28T15:08:55.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5b2c06c530 0x7f5b2c06e9f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:55.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b4c154dc0 0x7f5b4c150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:55.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b4c0cb180 0x7f5b4c1e7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:55.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 --2- 172.21.15.186:0/531581786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b4c0c52f0 0x7f5b4c151320 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:55.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 -- 172.21.15.186:0/531581786 >> 172.21.15.186:0/531581786 conn(0x7f5b4c0d00f0 msgr2=0x7f5b4c0b2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:55.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 -- 172.21.15.186:0/531581786 shutdown_connections 2024-03-28T15:08:55.679 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:55.678+0000 7f5b5088a640 1 -- 172.21.15.186:0/531581786 wait complete. 2024-03-28T15:08:55.690 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-03-28T15:08:56.988 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:08:56.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:08:57.402 INFO:tasks.ceph.mds.f.smithi191.stderr:2024-03-28T15:08:57.401+0000 7f7b08340640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T15:08:57.691 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:08:57.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.805+0000 7fa307102640 1 Processor -- start 2024-03-28T15:08:57.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.805+0000 7fa307102640 1 -- start start 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.806+0000 7fa307102640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30006f0a0 0x7fa300163f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.806+0000 7fa307102640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa30006f770 con 0x7fa30006d670 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.806+0000 7fa307102640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa300159500 con 0x7fa30006fb90 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.806+0000 7fa307102640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa3000c91a0 con 0x7fa30006f0a0 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.806+0000 7fa304e9e640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fa30006d670 0x7fa3000c8a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:55066/0 (socket says 172.21.15.186:55066) 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.806+0000 7fa304e9e640 1 -- 172.21.15.186:0/1240294004 learned_addr learned my addr 172.21.15.186:0/1240294004 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.806+0000 7fa30569f640 1 --2- 172.21.15.186:0/1240294004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30006f0a0 0x7fa300163f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.807+0000 7fa2ff7fe640 1 -- 172.21.15.186:0/1240294004 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3392649223 0 0) 0x7fa30006f770 con 0x7fa30006d670 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.807+0000 7fa2ff7fe640 1 -- 172.21.15.186:0/1240294004 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2f0003520 con 0x7fa30006d670 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.807+0000 7fa2ff7fe640 1 -- 172.21.15.186:0/1240294004 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3799371876 0 0) 0x7fa300159500 con 0x7fa30006fb90 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.807+0000 7fa2ff7fe640 1 -- 172.21.15.186:0/1240294004 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa30006f770 con 0x7fa30006fb90 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.807+0000 7fa30569f640 1 -- 172.21.15.186:0/1240294004 >> v1:172.21.15.191:6789/0 conn(0x7fa30006fb90 legacy=0x7fa30006da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.807+0000 7fa30569f640 1 -- 172.21.15.186:0/1240294004 >> v1:172.21.15.186:6789/0 conn(0x7fa30006d670 legacy=0x7fa3000c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:57.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.807+0000 7fa30569f640 1 -- 172.21.15.186:0/1240294004 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa300159220 con 0x7fa30006f0a0 2024-03-28T15:08:57.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.808+0000 7fa30569f640 1 --2- 172.21.15.186:0/1240294004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30006f0a0 0x7fa300163f90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8009dc0 tx=0x7fa2f802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b05bd852fa41c315 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:57.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.808+0000 7fa2ff7fe640 1 -- 172.21.15.186:0/1240294004 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8039070 con 0x7fa30006f0a0 2024-03-28T15:08:57.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.808+0000 7fa2ff7fe640 1 -- 172.21.15.186:0/1240294004 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2f802cc40 con 0x7fa30006f0a0 2024-03-28T15:08:57.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.808+0000 7fa2ff7fe640 1 -- 172.21.15.186:0/1240294004 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8034440 con 0x7fa30006f0a0 2024-03-28T15:08:57.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.808+0000 7fa307102640 1 -- 172.21.15.186:0/1240294004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30006f0a0 msgr2=0x7fa300163f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:57.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.808+0000 7fa307102640 1 --2- 172.21.15.186:0/1240294004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30006f0a0 0x7fa300163f90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8009dc0 tx=0x7fa2f802be90 comp rx=0 tx=0).stop 2024-03-28T15:08:57.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.809+0000 7fa307102640 1 -- 172.21.15.186:0/1240294004 shutdown_connections 2024-03-28T15:08:57.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.809+0000 7fa307102640 1 --2- 172.21.15.186:0/1240294004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30006f0a0 0x7fa300163f90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:57.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.809+0000 7fa307102640 1 -- 172.21.15.186:0/1240294004 >> 172.21.15.186:0/1240294004 conn(0x7fa300067670 msgr2=0x7fa300071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:57.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.809+0000 7fa307102640 1 -- 172.21.15.186:0/1240294004 shutdown_connections 2024-03-28T15:08:57.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.809+0000 7fa307102640 1 -- 172.21.15.186:0/1240294004 wait complete. 2024-03-28T15:08:57.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.810+0000 7fa307102640 1 Processor -- start 2024-03-28T15:08:57.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.810+0000 7fa307102640 1 -- start start 2024-03-28T15:08:57.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa307102640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30006d670 0x7fa3001f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:57.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa307102640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa30006f0a0 0x7fa3001433d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:57.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa307102640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30006fb90 0x7fa300156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:57.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa307102640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa3000c34c0 con 0x7fa30006f0a0 2024-03-28T15:08:57.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa307102640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa3000c4c60 con 0x7fa30006fb90 2024-03-28T15:08:57.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa307102640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa3000c50f0 con 0x7fa30006d670 2024-03-28T15:08:57.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa2fffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa30006f0a0 0x7fa3001433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:57.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa2fffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa30006f0a0 0x7fa3001433d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37064/0 (socket says 172.21.15.186:37064) 2024-03-28T15:08:57.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa304e9e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30006d670 0x7fa3001f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:57.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa2fffff640 1 -- 172.21.15.186:0/2523095718 learned_addr learned my addr 172.21.15.186:0/2523095718 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa30569f640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30006fb90 0x7fa300156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa2fffff640 1 -- 172.21.15.186:0/2523095718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30006d670 msgr2=0x7fa3001f4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.811+0000 7fa2fffff640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30006d670 0x7fa3001f4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.812+0000 7fa2fffff640 1 -- 172.21.15.186:0/2523095718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30006fb90 msgr2=0x7fa300156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.812+0000 7fa2fffff640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30006fb90 0x7fa300156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.812+0000 7fa2fffff640 1 -- 172.21.15.186:0/2523095718 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3001e2af0 con 0x7fa30006f0a0 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.812+0000 7fa2fffff640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa30006f0a0 0x7fa3001433d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c430 tx=0x7fa2ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.812+0000 7fa2fd7fa640 1 -- 172.21.15.186:0/2523095718 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec017070 con 0x7fa30006f0a0 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.812+0000 7fa2fd7fa640 1 -- 172.21.15.186:0/2523095718 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ec0052f0 con 0x7fa30006f0a0 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.812+0000 7fa307102640 1 -- 172.21.15.186:0/2523095718 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30006ce70 con 0x7fa30006f0a0 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.812+0000 7fa2fd7fa640 1 -- 172.21.15.186:0/2523095718 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec003830 con 0x7fa30006f0a0 2024-03-28T15:08:57.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.812+0000 7fa307102640 1 -- 172.21.15.186:0/2523095718 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa30006d430 con 0x7fa30006f0a0 2024-03-28T15:08:57.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.813+0000 7fa307102640 1 -- 172.21.15.186:0/2523095718 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3000686d0 con 0x7fa30006f0a0 2024-03-28T15:08:57.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.814+0000 7fa2fd7fa640 1 -- 172.21.15.186:0/2523095718 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa2ec01a070 con 0x7fa30006f0a0 2024-03-28T15:08:57.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.814+0000 7fa2fd7fa640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2e406c530 0x7fa2e406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:08:57.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.814+0000 7fa2fd7fa640 1 -- 172.21.15.186:0/2523095718 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7fa2ec08abf0 con 0x7fa30006f0a0 2024-03-28T15:08:57.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.814+0000 7fa304e9e640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2e406c530 0x7fa2e406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:08:57.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.815+0000 7fa304e9e640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2e406c530 0x7fa2e406e9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4000f60 tx=0x7fa2f4007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:08:57.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:57.820+0000 7fa2fd7fa640 1 -- 172.21.15.186:0/2523095718 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa2ec0589f0 con 0x7fa30006f0a0 2024-03-28T15:08:58.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.086+0000 7fa307102640 1 -- 172.21.15.186:0/2523095718 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3000c83d0 con 0x7fa30006f0a0 2024-03-28T15:08:58.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.086+0000 7fa2fd7fa640 1 -- 172.21.15.186:0/2523095718 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6848 (secure 0 0 0) 0x7fa2ec05ccf0 con 0x7fa30006f0a0 2024-03-28T15:08:58.087 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:08:58.087 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:49.855657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:active","state_seq":38,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5137,"qdb_cluster":[5137]},"id":1}]} 2024-03-28T15:08:58.087 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 29 2024-03-28T15:08:58.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 -- 172.21.15.186:0/2523095718 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2e406c530 msgr2=0x7fa2e406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:58.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2e406c530 0x7fa2e406e9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4000f60 tx=0x7fa2f4007d50 comp rx=0 tx=0).stop 2024-03-28T15:08:58.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 -- 172.21.15.186:0/2523095718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa30006f0a0 msgr2=0x7fa3001433d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:08:58.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa30006f0a0 0x7fa3001433d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c430 tx=0x7fa2ec00c8f0 comp rx=0 tx=0).stop 2024-03-28T15:08:58.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 -- 172.21.15.186:0/2523095718 shutdown_connections 2024-03-28T15:08:58.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa2e406c530 0x7fa2e406e9f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:58.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30006fb90 0x7fa300156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:58.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa30006f0a0 0x7fa3001433d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:58.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 --2- 172.21.15.186:0/2523095718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30006d670 0x7fa3001f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:08:58.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 -- 172.21.15.186:0/2523095718 >> 172.21.15.186:0/2523095718 conn(0x7fa300067670 msgr2=0x7fa30014bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:08:58.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 -- 172.21.15.186:0/2523095718 shutdown_connections 2024-03-28T15:08:58.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:08:58.089+0000 7fa307102640 1 -- 172.21.15.186:0/2523095718 wait complete. 2024-03-28T15:08:58.101 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-03-28T15:08:58.101 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2024-03-28T15:09:27.162 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:09:27.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:09:41.103 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:09:41.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.219+0000 7ff933db6640 1 Processor -- start 2024-03-28T15:09:41.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.219+0000 7ff933db6640 1 -- start start 2024-03-28T15:09:41.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.220+0000 7ff933db6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff92c1580b0 0x7ff92c0c8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.220+0000 7ff933db6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c0732e0 con 0x7ff92c0c8540 2024-03-28T15:09:41.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.220+0000 7ff933db6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c073b40 con 0x7ff92c0c7a10 2024-03-28T15:09:41.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.220+0000 7ff933db6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7ff92c072e50 con 0x7ff92c1580b0 2024-03-28T15:09:41.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.220+0000 7ff931351640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff92c0c8540 0x7ff92c06c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:38550/0 (socket says 172.21.15.186:38550) 2024-03-28T15:09:41.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.220+0000 7ff931351640 1 -- 172.21.15.186:0/2033825801 learned_addr learned my addr 172.21.15.186:0/2033825801 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:09:41.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.220+0000 7ff931b52640 1 --2- 172.21.15.186:0/2033825801 >> v2:172.21.15.191:3300/0 conn(0x7ff92c1580b0 0x7ff92c0c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.220+0000 7ff930b50640 1 -- 172.21.15.186:0/2033825801 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293216488 0 0) 0x7ff92c0732e0 con 0x7ff92c0c8540 2024-03-28T15:09:41.222 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.220+0000 7ff930b50640 1 -- 172.21.15.186:0/2033825801 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff918003520 con 0x7ff92c0c8540 2024-03-28T15:09:41.222 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.220+0000 7ff931b52640 1 -- 172.21.15.186:0/2033825801 >> v1:172.21.15.191:6789/0 conn(0x7ff92c0c7a10 legacy=0x7ff92c1650f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.222 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.220+0000 7ff931b52640 1 -- 172.21.15.186:0/2033825801 >> v1:172.21.15.186:6789/0 conn(0x7ff92c0c8540 legacy=0x7ff92c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.222 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.221+0000 7ff931b52640 1 -- 172.21.15.186:0/2033825801 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c06d860 con 0x7ff92c1580b0 2024-03-28T15:09:41.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.221+0000 7ff931b52640 1 --2- 172.21.15.186:0/2033825801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1580b0 0x7ff92c0c8000 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff9280097a0 tx=0x7ff92802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=771bdac4af01ce5e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:41.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.221+0000 7ff930b50640 1 -- 172.21.15.186:0/2033825801 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff928034070 con 0x7ff92c1580b0 2024-03-28T15:09:41.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.222+0000 7ff930b50640 1 -- 172.21.15.186:0/2033825801 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff928030610 con 0x7ff92c1580b0 2024-03-28T15:09:41.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.222+0000 7ff930b50640 1 -- 172.21.15.186:0/2033825801 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9280309c0 con 0x7ff92c1580b0 2024-03-28T15:09:41.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.222+0000 7ff933db6640 1 -- 172.21.15.186:0/2033825801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1580b0 msgr2=0x7ff92c0c8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.222+0000 7ff933db6640 1 --2- 172.21.15.186:0/2033825801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1580b0 0x7ff92c0c8000 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff9280097a0 tx=0x7ff92802b960 comp rx=0 tx=0).stop 2024-03-28T15:09:41.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.222+0000 7ff933db6640 1 -- 172.21.15.186:0/2033825801 shutdown_connections 2024-03-28T15:09:41.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.222+0000 7ff933db6640 1 --2- 172.21.15.186:0/2033825801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1580b0 0x7ff92c0c8000 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.222+0000 7ff933db6640 1 -- 172.21.15.186:0/2033825801 >> 172.21.15.186:0/2033825801 conn(0x7ff92c0c9b10 msgr2=0x7ff92c0c9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:41.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.222+0000 7ff933db6640 1 -- 172.21.15.186:0/2033825801 shutdown_connections 2024-03-28T15:09:41.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.222+0000 7ff933db6640 1 -- 172.21.15.186:0/2033825801 wait complete. 2024-03-28T15:09:41.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff933db6640 1 Processor -- start 2024-03-28T15:09:41.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff933db6640 1 -- start start 2024-03-28T15:09:41.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff933db6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c0c7a10 0x7ff92c148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff933db6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff92c0c8540 0x7ff92c1542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff933db6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1580b0 0x7ff92c1547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff933db6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c073580 con 0x7ff92c0c8540 2024-03-28T15:09:41.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff933db6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c073e20 con 0x7ff92c1580b0 2024-03-28T15:09:41.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff933db6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff92c073700 con 0x7ff92c0c7a10 2024-03-28T15:09:41.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff931351640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff92c0c8540 0x7ff92c1542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff931b52640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c0c7a10 0x7ff92c148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff931351640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff92c0c8540 0x7ff92c1542a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:46600/0 (socket says 172.21.15.186:46600) 2024-03-28T15:09:41.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff931351640 1 -- 172.21.15.186:0/2307905029 learned_addr learned my addr 172.21.15.186:0/2307905029 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:09:41.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.224+0000 7ff932353640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1580b0 0x7ff92c1547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff931351640 1 -- 172.21.15.186:0/2307905029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c0c7a10 msgr2=0x7ff92c148ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff931351640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c0c7a10 0x7ff92c148ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff931351640 1 -- 172.21.15.186:0/2307905029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1580b0 msgr2=0x7ff92c1547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff931351640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1580b0 0x7ff92c1547e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff931351640 1 -- 172.21.15.186:0/2307905029 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c1f3f40 con 0x7ff92c0c8540 2024-03-28T15:09:41.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff932353640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1580b0 0x7ff92c1547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:09:41.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff931351640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff92c0c8540 0x7ff92c1542a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff91c00c500 tx=0x7ff91c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:41.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff9227fc640 1 -- 172.21.15.186:0/2307905029 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c017070 con 0x7ff92c0c8540 2024-03-28T15:09:41.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff9227fc640 1 -- 172.21.15.186:0/2307905029 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff91c005ce0 con 0x7ff92c0c8540 2024-03-28T15:09:41.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff933db6640 1 -- 172.21.15.186:0/2307905029 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff92c1f4d60 con 0x7ff92c0c8540 2024-03-28T15:09:41.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff9227fc640 1 -- 172.21.15.186:0/2307905029 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c007330 con 0x7ff92c0c8540 2024-03-28T15:09:41.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.225+0000 7ff933db6640 1 -- 172.21.15.186:0/2307905029 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff92c1f44c0 con 0x7ff92c0c8540 2024-03-28T15:09:41.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.226+0000 7ff933db6640 1 -- 172.21.15.186:0/2307905029 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff92c0c3630 con 0x7ff92c0c8540 2024-03-28T15:09:41.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.227+0000 7ff9227fc640 1 -- 172.21.15.186:0/2307905029 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff91c01a070 con 0x7ff92c0c8540 2024-03-28T15:09:41.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.227+0000 7ff9227fc640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff91006c5b0 0x7ff91006ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.227+0000 7ff9227fc640 1 -- 172.21.15.186:0/2307905029 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7ff91c08ab40 con 0x7ff92c0c8540 2024-03-28T15:09:41.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.228+0000 7ff931b52640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff91006c5b0 0x7ff91006ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.228+0000 7ff931b52640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff91006c5b0 0x7ff91006ea70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff928036000 tx=0x7ff928008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:41.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.233+0000 7ff9227fc640 1 -- 172.21.15.186:0/2307905029 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff92c0c3630 con 0x7ff92c0c8540 2024-03-28T15:09:41.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.500+0000 7ff933db6640 1 -- 172.21.15.186:0/2307905029 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff92c0ccd60 con 0x7ff92c0c8540 2024-03-28T15:09:41.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.501+0000 7ff9227fc640 1 -- 172.21.15.186:0/2307905029 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6848 (secure 0 0 0) 0x7ff92c0ccd60 con 0x7ff92c0c8540 2024-03-28T15:09:41.501 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:09:41.501 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:49.855657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:active","state_seq":38,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5137,"qdb_cluster":[5137]},"id":1}]} 2024-03-28T15:09:41.502 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 29 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 -- 172.21.15.186:0/2307905029 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff91006c5b0 msgr2=0x7ff91006ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff91006c5b0 0x7ff91006ea70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff928036000 tx=0x7ff928008030 comp rx=0 tx=0).stop 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 -- 172.21.15.186:0/2307905029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff92c0c8540 msgr2=0x7ff92c1542a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff92c0c8540 0x7ff92c1542a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff91c00c500 tx=0x7ff91c00c9c0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 -- 172.21.15.186:0/2307905029 shutdown_connections 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff91006c5b0 0x7ff91006ea70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1580b0 0x7ff92c1547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff92c0c8540 0x7ff92c1542a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 --2- 172.21.15.186:0/2307905029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c0c7a10 0x7ff92c148ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 -- 172.21.15.186:0/2307905029 >> 172.21.15.186:0/2307905029 conn(0x7ff92c0c9b10 msgr2=0x7ff92c0c2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 -- 172.21.15.186:0/2307905029 shutdown_connections 2024-03-28T15:09:41.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.503+0000 7ff933db6640 1 -- 172.21.15.186:0/2307905029 wait complete. 2024-03-28T15:09:41.516 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2024-03-28T15:09:41.516 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-03-28T15:09:41.516 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:09:41.530 INFO:tasks.ceph.mds.f:Stopped 2024-03-28T15:09:41.530 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:09:41.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.630+0000 7fdd23f83640 1 Processor -- start 2024-03-28T15:09:41.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.630+0000 7fdd23f83640 1 -- start start 2024-03-28T15:09:41.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd23f83640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd1c151990 0x7fdd1c0c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd23f83640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdd1c0b1e30 0x7fdd1c0c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd23f83640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd1c06caa0 con 0x7fdd1c0c2d00 2024-03-28T15:09:41.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd23f83640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fdd1c0c1da0 con 0x7fdd1c0b1e30 2024-03-28T15:09:41.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd23f83640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1c06cd80 con 0x7fdd1c151990 2024-03-28T15:09:41.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd22520640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fdd1c0c2d00 0x7fdd1c0712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:38564/0 (socket says 172.21.15.186:38564) 2024-03-28T15:09:41.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd22520640 1 -- 172.21.15.186:0/2133284166 learned_addr learned my addr 172.21.15.186:0/2133284166 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:09:41.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd2151e640 1 --2- 172.21.15.186:0/2133284166 >> v2:172.21.15.191:3300/0 conn(0x7fdd1c0b1e30 0x7fdd1c0c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd21d1f640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd1c151990 0x7fdd1c0c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3800666722 0 0) 0x7fdd1c06caa0 con 0x7fdd1c0c2d00 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd08003520 con 0x7fdd1c0c2d00 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3824595838 0 0) 0x7fdd08003520 con 0x7fdd1c0c2d00 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd1c151990 msgr2=0x7fdd1c0c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd20d1d640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd1c151990 0x7fdd1c0c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 >> v2:172.21.15.191:3300/0 conn(0x7fdd1c0b1e30 msgr2=0x7fdd1c0c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd20d1d640 1 --2- 172.21.15.186:0/2133284166 >> v2:172.21.15.191:3300/0 conn(0x7fdd1c0b1e30 0x7fdd1c0c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1c0c1340 con 0x7fdd1c0c2d00 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd21d1f640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd1c151990 0x7fdd1c0c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7fdd140019f0 con 0x7fdd1c0c2d00 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd2151e640 1 --2- 172.21.15.186:0/2133284166 >> v2:172.21.15.191:3300/0 conn(0x7fdd1c0b1e30 0x7fdd1c0c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.631+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 >> v1:172.21.15.186:6789/0 conn(0x7fdd1c0c2d00 legacy=0x7fdd1c0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd20d1d640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd080039b0 0x7fdd08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd20d1d640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd08004800 0x7fdd08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd20d1d640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd1c151990 0x7fdd080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c0c1da0 con 0x7fdd080039b0 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c06cd80 con 0x7fdd08004800 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fdd0800b9c0 con 0x7fdd1c151990 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd21d1f640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd080039b0 0x7fdd08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd2151e640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd08004800 0x7fdd08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd22520640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd1c151990 0x7fdd080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd21d1f640 1 -- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd1c151990 msgr2=0x7fdd080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd21d1f640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd1c151990 0x7fdd080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd21d1f640 1 -- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd08004800 msgr2=0x7fdd08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd21d1f640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd08004800 0x7fdd08004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.632+0000 7fdd21d1f640 1 -- 172.21.15.186:0/2133284166 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd08003520 con 0x7fdd080039b0 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.633+0000 7fdd21d1f640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd080039b0 0x7fdd08003dd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009c90 tx=0x7fdd08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4aabfc0197d7402 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.633+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18039070 con 0x7fdd080039b0 2024-03-28T15:09:41.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.633+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd1802c8f0 con 0x7fdd080039b0 2024-03-28T15:09:41.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.633+0000 7fdd20d1d640 1 -- 172.21.15.186:0/2133284166 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18009860 con 0x7fdd080039b0 2024-03-28T15:09:41.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.633+0000 7fdd23f83640 1 -- 172.21.15.186:0/2133284166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd080039b0 msgr2=0x7fdd08003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.633+0000 7fdd23f83640 1 --2- 172.21.15.186:0/2133284166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd080039b0 0x7fdd08003dd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009c90 tx=0x7fdd08005830 comp rx=0 tx=0).stop 2024-03-28T15:09:41.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.633+0000 7fdd22520640 1 -- 172.21.15.186:0/2133284166 reap_dead start 2024-03-28T15:09:41.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.633+0000 7fdd23f83640 1 -- 172.21.15.186:0/2133284166 shutdown_connections 2024-03-28T15:09:41.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.633+0000 7fdd23f83640 1 -- 172.21.15.186:0/2133284166 >> 172.21.15.186:0/2133284166 conn(0x7fdd1c0c8a20 msgr2=0x7fdd1c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:41.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.633+0000 7fdd23f83640 1 -- 172.21.15.186:0/2133284166 shutdown_connections 2024-03-28T15:09:41.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.633+0000 7fdd23f83640 1 -- 172.21.15.186:0/2133284166 wait complete. 2024-03-28T15:09:41.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd23f83640 1 Processor -- start 2024-03-28T15:09:41.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd23f83640 1 -- start start 2024-03-28T15:09:41.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd23f83640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd080039b0 0x7fdd1c1424a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd23f83640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd1c153520 0x7fdd1c153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd23f83640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd1c154480 0x7fdd1c154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd23f83640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c073dd0 con 0x7fdd1c153520 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd23f83640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c0c1900 con 0x7fdd1c154480 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd23f83640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1c073f50 con 0x7fdd080039b0 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd2151e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd1c153520 0x7fdd1c153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd2151e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd1c153520 0x7fdd1c153940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:46626/0 (socket says 172.21.15.186:46626) 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd21d1f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd080039b0 0x7fdd1c1424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd22520640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd1c154480 0x7fdd1c154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.635+0000 7fdd2151e640 1 -- 172.21.15.186:0/2684167560 learned_addr learned my addr 172.21.15.186:0/2684167560 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.636+0000 7fdd2151e640 1 -- 172.21.15.186:0/2684167560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd080039b0 msgr2=0x7fdd1c1424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.636+0000 7fdd2151e640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd080039b0 0x7fdd1c1424a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.636+0000 7fdd2151e640 1 -- 172.21.15.186:0/2684167560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd1c154480 msgr2=0x7fdd1c154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.636+0000 7fdd2151e640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd1c154480 0x7fdd1c154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.636+0000 7fdd2151e640 1 -- 172.21.15.186:0/2684167560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1c154e70 con 0x7fdd1c153520 2024-03-28T15:09:41.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.636+0000 7fdd2151e640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd1c153520 0x7fdd1c153940 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c011cd0 tx=0x7fdd0c00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:41.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.636+0000 7fdd127fc640 1 -- 172.21.15.186:0/2684167560 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c011d00 con 0x7fdd1c153520 2024-03-28T15:09:41.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.636+0000 7fdd127fc640 1 -- 172.21.15.186:0/2684167560 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd0c014440 con 0x7fdd1c153520 2024-03-28T15:09:41.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.636+0000 7fdd23f83640 1 -- 172.21.15.186:0/2684167560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd1c0c50e0 con 0x7fdd1c153520 2024-03-28T15:09:41.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.636+0000 7fdd127fc640 1 -- 172.21.15.186:0/2684167560 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c011ea0 con 0x7fdd1c153520 2024-03-28T15:09:41.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.636+0000 7fdd23f83640 1 -- 172.21.15.186:0/2684167560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd1c0c41b0 con 0x7fdd1c153520 2024-03-28T15:09:41.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.637+0000 7fdd23f83640 1 -- 172.21.15.186:0/2684167560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd1c14dd50 con 0x7fdd1c153520 2024-03-28T15:09:41.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.637+0000 7fdd127fc640 1 -- 172.21.15.186:0/2684167560 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdd0c012040 con 0x7fdd1c153520 2024-03-28T15:09:41.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.638+0000 7fdd127fc640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdd0006c560 0x7fdd0006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:41.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.638+0000 7fdd127fc640 1 -- 172.21.15.186:0/2684167560 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7fdd0c08be90 con 0x7fdd1c153520 2024-03-28T15:09:41.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.638+0000 7fdd21d1f640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdd0006c560 0x7fdd0006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:41.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.638+0000 7fdd21d1f640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdd0006c560 0x7fdd0006ea20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdd18036000 tx=0x7fdd18030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:41.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.642+0000 7fdd127fc640 1 -- 172.21.15.186:0/2684167560 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fdd0c019030 con 0x7fdd1c153520 2024-03-28T15:09:41.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.855+0000 7fdd23f83640 1 -- 172.21.15.186:0/2684167560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd1c1583e0 con 0x7fdd1c153520 2024-03-28T15:09:41.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.856+0000 7fdd127fc640 1 -- 172.21.15.186:0/2684167560 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6848 (secure 0 0 0) 0x7fdd0c05a2e0 con 0x7fdd1c153520 2024-03-28T15:09:41.857 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:09:41.857 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:49.855657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:active","state_seq":38,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5137,"qdb_cluster":[5137]},"id":1}]} 2024-03-28T15:09:41.857 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 29 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 -- 172.21.15.186:0/2684167560 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdd0006c560 msgr2=0x7fdd0006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdd0006c560 0x7fdd0006ea20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdd18036000 tx=0x7fdd18030000 comp rx=0 tx=0).stop 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 -- 172.21.15.186:0/2684167560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd1c153520 msgr2=0x7fdd1c153940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd1c153520 0x7fdd1c153940 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c011cd0 tx=0x7fdd0c00f610 comp rx=0 tx=0).stop 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 -- 172.21.15.186:0/2684167560 shutdown_connections 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdd0006c560 0x7fdd0006ea20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd1c154480 0x7fdd1c154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd1c153520 0x7fdd1c153940 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 --2- 172.21.15.186:0/2684167560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd080039b0 0x7fdd1c1424a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 -- 172.21.15.186:0/2684167560 >> 172.21.15.186:0/2684167560 conn(0x7fdd1c0c8a20 msgr2=0x7fdd1c1579d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 -- 172.21.15.186:0/2684167560 shutdown_connections 2024-03-28T15:09:41.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:41.858+0000 7fdd23f83640 1 -- 172.21.15.186:0/2684167560 wait complete. 2024-03-28T15:09:41.869 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:09:43.870 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:09:43.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.982+0000 7f9a4eb7a640 1 Processor -- start 2024-03-28T15:09:43.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.982+0000 7f9a4eb7a640 1 -- start start 2024-03-28T15:09:43.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.983+0000 7f9a4eb7a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9a48071340 0x7f9a48158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:43.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.983+0000 7f9a4eb7a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a480c8090 0x7f9a480c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:43.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.983+0000 7f9a4eb7a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a48072df0 con 0x7f9a48152810 2024-03-28T15:09:43.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.983+0000 7f9a4eb7a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f9a48159500 con 0x7f9a48071340 2024-03-28T15:09:43.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.983+0000 7f9a4eb7a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f9a480c8660 con 0x7f9a480c8090 2024-03-28T15:09:43.984 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.983+0000 7f9a47fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9a48071340 0x7f9a48158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:43.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.983+0000 7f9a47fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9a48071340 0x7f9a48158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:46634/0 (socket says 172.21.15.186:46634) 2024-03-28T15:09:43.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.983+0000 7f9a47fff640 1 -- 172.21.15.186:0/3054613614 learned_addr learned my addr 172.21.15.186:0/3054613614 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:09:43.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.983+0000 7f9a4d117640 1 --2- 172.21.15.186:0/3054613614 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a480c8090 0x7f9a480c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:43.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.984+0000 7f9a47fff640 1 -- 172.21.15.186:0/3054613614 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a480c8090 msgr2=0x7f9a480c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:09:43.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.984+0000 7f9a47fff640 1 --2- 172.21.15.186:0/3054613614 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a480c8090 0x7f9a480c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:43.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.984+0000 7f9a47fff640 1 -- 172.21.15.186:0/3054613614 >> v1:172.21.15.191:6789/0 conn(0x7f9a48152810 legacy=0x7f9a4806c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:43.985 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.984+0000 7f9a47fff640 1 -- 172.21.15.186:0/3054613614 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a48071d50 con 0x7f9a48071340 2024-03-28T15:09:43.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.984+0000 7f9a47fff640 1 --2- 172.21.15.186:0/3054613614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a48071340 0x7f9a48158fc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009870 tx=0x7f9a3c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8f111e6d1e4eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:43.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.985+0000 7f9a477fe640 1 -- 172.21.15.186:0/3054613614 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c034070 con 0x7f9a48071340 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.985+0000 7f9a477fe640 1 -- 172.21.15.186:0/3054613614 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a3c030840 con 0x7f9a48071340 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.985+0000 7f9a477fe640 1 -- 172.21.15.186:0/3054613614 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c030c10 con 0x7f9a48071340 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.985+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/3054613614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a48071340 msgr2=0x7f9a48158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.985+0000 7f9a4eb7a640 1 --2- 172.21.15.186:0/3054613614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a48071340 0x7f9a48158fc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009870 tx=0x7f9a3c02b980 comp rx=0 tx=0).stop 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.985+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/3054613614 shutdown_connections 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.985+0000 7f9a4eb7a640 1 --2- 172.21.15.186:0/3054613614 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a480c8090 0x7f9a480c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.985+0000 7f9a4eb7a640 1 --2- 172.21.15.186:0/3054613614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a48071340 0x7f9a48158fc0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.985+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/3054613614 >> 172.21.15.186:0/3054613614 conn(0x7f9a480c8a20 msgr2=0x7f9a480c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.985+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/3054613614 shutdown_connections 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.985+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/3054613614 wait complete. 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.986+0000 7f9a4eb7a640 1 Processor -- start 2024-03-28T15:09:43.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.986+0000 7f9a4eb7a640 1 -- start start 2024-03-28T15:09:43.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.987+0000 7f9a4eb7a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a48071340 0x7f9a48144fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:43.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.987+0000 7f9a4eb7a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a480c8090 0x7f9a4814cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:43.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.987+0000 7f9a4eb7a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a48152810 0x7f9a4814d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.987+0000 7f9a4eb7a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a480725f0 con 0x7f9a48152810 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.987+0000 7f9a4eb7a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9a48072310 con 0x7f9a480c8090 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.987+0000 7f9a4eb7a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f9a480728d0 con 0x7f9a48071340 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.987+0000 7f9a4d117640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a48152810 0x7f9a4814d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.987+0000 7f9a4d117640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a48152810 0x7f9a4814d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:46650/0 (socket says 172.21.15.186:46650) 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.987+0000 7f9a4d117640 1 -- 172.21.15.186:0/1976846735 learned_addr learned my addr 172.21.15.186:0/1976846735 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.987+0000 7f9a47fff640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a480c8090 0x7f9a4814cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.987+0000 7f9a4c916640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a48071340 0x7f9a48144fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.988+0000 7f9a4d117640 1 -- 172.21.15.186:0/1976846735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a48071340 msgr2=0x7f9a48144fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.988+0000 7f9a4d117640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a48071340 0x7f9a48144fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.988+0000 7f9a4d117640 1 -- 172.21.15.186:0/1976846735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a480c8090 msgr2=0x7f9a4814cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.988+0000 7f9a4d117640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a480c8090 0x7f9a4814cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.988+0000 7f9a4d117640 1 -- 172.21.15.186:0/1976846735 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a481454e0 con 0x7f9a48152810 2024-03-28T15:09:43.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.988+0000 7f9a4d117640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a48152810 0x7f9a4814d000 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009dd0 tx=0x7f9a4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:43.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.988+0000 7f9a457fa640 1 -- 172.21.15.186:0/1976846735 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40018020 con 0x7f9a48152810 2024-03-28T15:09:43.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.988+0000 7f9a457fa640 1 -- 172.21.15.186:0/1976846735 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a40014480 con 0x7f9a48152810 2024-03-28T15:09:43.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.988+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/1976846735 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a480c2ae0 con 0x7f9a48152810 2024-03-28T15:09:43.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.988+0000 7f9a457fa640 1 -- 172.21.15.186:0/1976846735 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40014870 con 0x7f9a48152810 2024-03-28T15:09:43.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.988+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/1976846735 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a481530e0 con 0x7f9a48152810 2024-03-28T15:09:43.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.989+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/1976846735 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a48069500 con 0x7f9a48152810 2024-03-28T15:09:43.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.989+0000 7f9a457fa640 1 -- 172.21.15.186:0/1976846735 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9a40005d10 con 0x7f9a48152810 2024-03-28T15:09:43.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.990+0000 7f9a457fa640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9a3406c530 0x7f9a3406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:43.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.990+0000 7f9a457fa640 1 -- 172.21.15.186:0/1976846735 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7f9a4008c240 con 0x7f9a48152810 2024-03-28T15:09:43.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.990+0000 7f9a4c916640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9a3406c530 0x7f9a3406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:43.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.990+0000 7f9a4c916640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9a3406c530 0x7f9a3406e9f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9a380016e0 tx=0x7f9a38009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:43.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:43.994+0000 7f9a457fa640 1 -- 172.21.15.186:0/1976846735 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f9a40022070 con 0x7f9a48152810 2024-03-28T15:09:44.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.207+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/1976846735 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a480c0f40 con 0x7f9a48152810 2024-03-28T15:09:44.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.207+0000 7f9a457fa640 1 -- 172.21.15.186:0/1976846735 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6848 (secure 0 0 0) 0x7f9a40010040 con 0x7f9a48152810 2024-03-28T15:09:44.208 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:09:44.208 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:49.855657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:active","state_seq":38,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5137,"qdb_cluster":[5137]},"id":1}]} 2024-03-28T15:09:44.209 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 29 2024-03-28T15:09:44.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.209+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/1976846735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9a3406c530 msgr2=0x7f9a3406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:44.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.209+0000 7f9a4eb7a640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9a3406c530 0x7f9a3406e9f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9a380016e0 tx=0x7f9a38009040 comp rx=0 tx=0).stop 2024-03-28T15:09:44.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.209+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/1976846735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a48152810 msgr2=0x7f9a4814d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:44.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.209+0000 7f9a4eb7a640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a48152810 0x7f9a4814d000 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009dd0 tx=0x7f9a4000c490 comp rx=0 tx=0).stop 2024-03-28T15:09:44.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.209+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/1976846735 shutdown_connections 2024-03-28T15:09:44.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.209+0000 7f9a4eb7a640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f9a3406c530 0x7f9a3406e9f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:44.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.209+0000 7f9a4eb7a640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a48152810 0x7f9a4814d000 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:44.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.209+0000 7f9a4eb7a640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a480c8090 0x7f9a4814cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:44.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.209+0000 7f9a4eb7a640 1 --2- 172.21.15.186:0/1976846735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a48071340 0x7f9a48144fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:44.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.210+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/1976846735 >> 172.21.15.186:0/1976846735 conn(0x7f9a480c8a20 msgr2=0x7f9a480c4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:44.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.210+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/1976846735 shutdown_connections 2024-03-28T15:09:44.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:44.210+0000 7f9a4eb7a640 1 -- 172.21.15.186:0/1976846735 wait complete. 2024-03-28T15:09:44.221 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:09:46.221 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:09:46.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.336+0000 7fd8d3ef4640 1 Processor -- start 2024-03-28T15:09:46.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.336+0000 7fd8d3ef4640 1 -- start start 2024-03-28T15:09:46.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.337+0000 7fd8d3ef4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd8cc06faf0 0x7fd8cc0c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:46.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.337+0000 7fd8d3ef4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8cc154dd0 0x7fd8cc15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:46.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.337+0000 7fd8d3ef4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8cc0c71b0 con 0x7fd8cc06f710 2024-03-28T15:09:46.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.337+0000 7fd8d3ef4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fd8cc071bc0 con 0x7fd8cc06faf0 2024-03-28T15:09:46.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.337+0000 7fd8d3ef4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd8cc072660 con 0x7fd8cc154dd0 2024-03-28T15:09:46.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.338+0000 7fd8d2491640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8cc154dd0 0x7fd8cc15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:46.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.338+0000 7fd8d148f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd8cc06faf0 0x7fd8cc0c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:46.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.338+0000 7fd8d1c90640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd8cc06f710 0x7fd8cc1546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:36664/0 (socket says 172.21.15.186:36664) 2024-03-28T15:09:46.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.338+0000 7fd8d1c90640 1 -- 172.21.15.186:0/239110686 learned_addr learned my addr 172.21.15.186:0/239110686 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:09:46.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.338+0000 7fd8d2491640 1 -- 172.21.15.186:0/239110686 >> v2:172.21.15.191:3300/0 conn(0x7fd8cc06faf0 msgr2=0x7fd8cc0c6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:09:46.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.338+0000 7fd8d2491640 1 --2- 172.21.15.186:0/239110686 >> v2:172.21.15.191:3300/0 conn(0x7fd8cc06faf0 0x7fd8cc0c6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:46.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.338+0000 7fd8d2491640 1 -- 172.21.15.186:0/239110686 >> v1:172.21.15.191:6789/0 conn(0x7fd8cc06f710 legacy=0x7fd8cc1546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:46.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.338+0000 7fd8d2491640 1 -- 172.21.15.186:0/239110686 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8cc072900 con 0x7fd8cc154dd0 2024-03-28T15:09:46.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.338+0000 7fd8d148f640 1 --2- 172.21.15.186:0/239110686 >> v2:172.21.15.191:3300/0 conn(0x7fd8cc06faf0 0x7fd8cc0c6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:09:46.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.339+0000 7fd8d2491640 1 --2- 172.21.15.186:0/239110686 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8cc154dd0 0x7fd8cc15aba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009d90 tx=0x7fd8c402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b2148c2ffdb80e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:46.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.339+0000 7fd8d0c8e640 1 -- 172.21.15.186:0/239110686 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4039070 con 0x7fd8cc154dd0 2024-03-28T15:09:46.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.339+0000 7fd8d0c8e640 1 -- 172.21.15.186:0/239110686 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8c402cbc0 con 0x7fd8cc154dd0 2024-03-28T15:09:46.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.339+0000 7fd8d0c8e640 1 -- 172.21.15.186:0/239110686 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4034440 con 0x7fd8cc154dd0 2024-03-28T15:09:46.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.339+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/239110686 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8cc154dd0 msgr2=0x7fd8cc15aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:46.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.340+0000 7fd8d3ef4640 1 --2- 172.21.15.186:0/239110686 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8cc154dd0 0x7fd8cc15aba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009d90 tx=0x7fd8c402be10 comp rx=0 tx=0).stop 2024-03-28T15:09:46.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.340+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/239110686 shutdown_connections 2024-03-28T15:09:46.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.340+0000 7fd8d3ef4640 1 --2- 172.21.15.186:0/239110686 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8cc154dd0 0x7fd8cc15aba0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:46.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.340+0000 7fd8d3ef4640 1 --2- 172.21.15.186:0/239110686 >> v2:172.21.15.191:3300/0 conn(0x7fd8cc06faf0 0x7fd8cc0c6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:46.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.340+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/239110686 >> 172.21.15.186:0/239110686 conn(0x7fd8cc06d680 msgr2=0x7fd8cc06da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:46.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.340+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/239110686 shutdown_connections 2024-03-28T15:09:46.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.340+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/239110686 wait complete. 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.341+0000 7fd8d3ef4640 1 Processor -- start 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.341+0000 7fd8d3ef4640 1 -- start start 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.341+0000 7fd8d3ef4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8cc06f710 0x7fd8cc0c4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.341+0000 7fd8d3ef4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8cc06faf0 0x7fd8cc0cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.341+0000 7fd8d3ef4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8cc154dd0 0x7fd8cc14ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.341+0000 7fd8d3ef4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd8cc0731a0 con 0x7fd8cc06f710 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.341+0000 7fd8d3ef4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd8cc072ec0 con 0x7fd8cc154dd0 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.341+0000 7fd8d3ef4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd8cc0c8bb0 con 0x7fd8cc06faf0 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d1c90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8cc06f710 0x7fd8cc0c4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d1c90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8cc06f710 0x7fd8cc0c4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:46664/0 (socket says 172.21.15.186:46664) 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d1c90640 1 -- 172.21.15.186:0/2388938632 learned_addr learned my addr 172.21.15.186:0/2388938632 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:09:46.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d148f640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8cc06faf0 0x7fd8cc0cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:46.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d2491640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8cc154dd0 0x7fd8cc14ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:46.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d1c90640 1 -- 172.21.15.186:0/2388938632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8cc06faf0 msgr2=0x7fd8cc0cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:46.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d1c90640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8cc06faf0 0x7fd8cc0cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:46.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d1c90640 1 -- 172.21.15.186:0/2388938632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8cc154dd0 msgr2=0x7fd8cc14ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:46.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d1c90640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8cc154dd0 0x7fd8cc14ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:46.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d1c90640 1 -- 172.21.15.186:0/2388938632 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8cc14d480 con 0x7fd8cc06f710 2024-03-28T15:09:46.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d1c90640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8cc06f710 0x7fd8cc0c4370 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800c430 tx=0x7fd8c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:46.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8c27fc640 1 -- 172.21.15.186:0/2388938632 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8017070 con 0x7fd8cc06f710 2024-03-28T15:09:46.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8c27fc640 1 -- 172.21.15.186:0/2388938632 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8c8005ce0 con 0x7fd8cc06f710 2024-03-28T15:09:46.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/2388938632 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8cc0c48b0 con 0x7fd8cc06f710 2024-03-28T15:09:46.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8c27fc640 1 -- 172.21.15.186:0/2388938632 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c80072f0 con 0x7fd8cc06f710 2024-03-28T15:09:46.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.342+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/2388938632 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8cc1f1180 con 0x7fd8cc06f710 2024-03-28T15:09:46.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.343+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/2388938632 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8cc069500 con 0x7fd8cc06f710 2024-03-28T15:09:46.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.344+0000 7fd8c27fc640 1 -- 172.21.15.186:0/2388938632 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd8c801a070 con 0x7fd8cc06f710 2024-03-28T15:09:46.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.344+0000 7fd8c27fc640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8b006c530 0x7fd8b006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:46.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.344+0000 7fd8c27fc640 1 -- 172.21.15.186:0/2388938632 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7fd8c808abd0 con 0x7fd8cc06f710 2024-03-28T15:09:46.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.344+0000 7fd8d148f640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8b006c530 0x7fd8b006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:46.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.345+0000 7fd8d148f640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8b006c530 0x7fd8b006e9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009560 tx=0x7fd8bc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:46.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.349+0000 7fd8c27fc640 1 -- 172.21.15.186:0/2388938632 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd8c80589d0 con 0x7fd8cc06f710 2024-03-28T15:09:46.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.563+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/2388938632 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8cc154a50 con 0x7fd8cc06f710 2024-03-28T15:09:46.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.563+0000 7fd8c27fc640 1 -- 172.21.15.186:0/2388938632 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6848 (secure 0 0 0) 0x7fd8c805ccd0 con 0x7fd8cc06f710 2024-03-28T15:09:46.565 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:09:46.565 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:49.855657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:active","state_seq":38,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5137,"qdb_cluster":[5137]},"id":1}]} 2024-03-28T15:09:46.566 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 29 2024-03-28T15:09:46.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/2388938632 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8b006c530 msgr2=0x7fd8b006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:46.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8b006c530 0x7fd8b006e9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009560 tx=0x7fd8bc009190 comp rx=0 tx=0).stop 2024-03-28T15:09:46.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/2388938632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8cc06f710 msgr2=0x7fd8cc0c4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:46.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8cc06f710 0x7fd8cc0c4370 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800c430 tx=0x7fd8c800c8f0 comp rx=0 tx=0).stop 2024-03-28T15:09:46.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/2388938632 shutdown_connections 2024-03-28T15:09:46.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8b006c530 0x7fd8b006e9f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:46.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8cc154dd0 0x7fd8cc14ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:46.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8cc06faf0 0x7fd8cc0cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:46.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 --2- 172.21.15.186:0/2388938632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8cc06f710 0x7fd8cc0c4370 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:46.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/2388938632 >> 172.21.15.186:0/2388938632 conn(0x7fd8cc06d680 msgr2=0x7fd8cc153970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:46.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/2388938632 shutdown_connections 2024-03-28T15:09:46.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:46.565+0000 7fd8d3ef4640 1 -- 172.21.15.186:0/2388938632 wait complete. 2024-03-28T15:09:46.577 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:09:48.579 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:09:48.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.690+0000 7f99f8b63640 1 Processor -- start 2024-03-28T15:09:48.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.690+0000 7f99f8b63640 1 -- start start 2024-03-28T15:09:48.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.691+0000 7f99f8b63640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f99f4154bd0 0x7f99f415a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:48.692 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.691+0000 7f99f8b63640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99f40c6e00 con 0x7f99f40c53c0 2024-03-28T15:09:48.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.691+0000 7f99f8b63640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99f415b080 con 0x7f99f40c57e0 2024-03-28T15:09:48.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.691+0000 7f99f8b63640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f99f40c5ca0 con 0x7f99f4154bd0 2024-03-28T15:09:48.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.691+0000 7f99f1d9b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f99f40c53c0 0x7f99f40c66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:50534/0 (socket says 172.21.15.186:50534) 2024-03-28T15:09:48.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.692+0000 7f99f1d9b640 1 -- 172.21.15.186:0/597493213 learned_addr learned my addr 172.21.15.186:0/597493213 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:09:48.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.692+0000 7f99f259c640 1 --2- 172.21.15.186:0/597493213 >> v2:172.21.15.191:3300/0 conn(0x7f99f4154bd0 0x7f99f415a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:48.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.692+0000 7f99f159a640 1 -- 172.21.15.186:0/597493213 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2650348541 0 0) 0x7f99f40c6e00 con 0x7f99f40c53c0 2024-03-28T15:09:48.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.692+0000 7f99f159a640 1 -- 172.21.15.186:0/597493213 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99e0003520 con 0x7f99f40c53c0 2024-03-28T15:09:48.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.692+0000 7f99f159a640 1 -- 172.21.15.186:0/597493213 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1291283649 0 0) 0x7f99f415b080 con 0x7f99f40c57e0 2024-03-28T15:09:48.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.692+0000 7f99f159a640 1 -- 172.21.15.186:0/597493213 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99f40c6e00 con 0x7f99f40c57e0 2024-03-28T15:09:48.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.692+0000 7f99f259c640 1 -- 172.21.15.186:0/597493213 >> v1:172.21.15.191:6789/0 conn(0x7f99f40c57e0 legacy=0x7f99f40cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:48.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.692+0000 7f99f259c640 1 -- 172.21.15.186:0/597493213 >> v1:172.21.15.186:6789/0 conn(0x7f99f40c53c0 legacy=0x7f99f40c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:48.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.692+0000 7f99f159a640 1 -- 172.21.15.186:0/597493213 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3172717313 0 0) 0x7f99e0003520 con 0x7f99f40c53c0 2024-03-28T15:09:48.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.692+0000 7f99f259c640 1 -- 172.21.15.186:0/597493213 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f415b360 con 0x7f99f4154bd0 2024-03-28T15:09:48.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.693+0000 7f99f259c640 1 --2- 172.21.15.186:0/597493213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99f4154bd0 0x7f99f415a7d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009d90 tx=0x7f99e402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d31d2edef5c851 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:48.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.693+0000 7f99f159a640 1 -- 172.21.15.186:0/597493213 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4039070 con 0x7f99f4154bd0 2024-03-28T15:09:48.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.693+0000 7f99f159a640 1 -- 172.21.15.186:0/597493213 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99e402c9d0 con 0x7f99f4154bd0 2024-03-28T15:09:48.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.693+0000 7f99f159a640 1 -- 172.21.15.186:0/597493213 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99dc001670 con 0x7f99f4154bd0 2024-03-28T15:09:48.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.694+0000 7f99f8b63640 1 -- 172.21.15.186:0/597493213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99f4154bd0 msgr2=0x7f99f415a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:48.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.694+0000 7f99f8b63640 1 --2- 172.21.15.186:0/597493213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99f4154bd0 0x7f99f415a7d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009d90 tx=0x7f99e402bea0 comp rx=0 tx=0).stop 2024-03-28T15:09:48.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.694+0000 7f99f8b63640 1 -- 172.21.15.186:0/597493213 shutdown_connections 2024-03-28T15:09:48.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.694+0000 7f99f8b63640 1 --2- 172.21.15.186:0/597493213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99f4154bd0 0x7f99f415a7d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:48.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.694+0000 7f99f8b63640 1 -- 172.21.15.186:0/597493213 >> 172.21.15.186:0/597493213 conn(0x7f99f40731f0 msgr2=0x7f99f4073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:48.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.694+0000 7f99f8b63640 1 -- 172.21.15.186:0/597493213 shutdown_connections 2024-03-28T15:09:48.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.694+0000 7f99f8b63640 1 -- 172.21.15.186:0/597493213 wait complete. 2024-03-28T15:09:48.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.695+0000 7f99f8b63640 1 Processor -- start 2024-03-28T15:09:48.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.695+0000 7f99f8b63640 1 -- start start 2024-03-28T15:09:48.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f8b63640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f40c53c0 0x7f99f415a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:48.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f8b63640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99f40c57e0 0x7f99f40b5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:48.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f8b63640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99f4154bd0 0x7f99f41e5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:48.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f8b63640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f99f40c61d0 con 0x7f99f40c53c0 2024-03-28T15:09:48.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f8b63640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f99f40c5ef0 con 0x7f99f40c57e0 2024-03-28T15:09:48.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f8b63640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f99f40cc390 con 0x7f99f4154bd0 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f40c53c0 0x7f99f415a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f40c53c0 0x7f99f415a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:38294/0 (socket says 172.21.15.186:38294) 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f259c640 1 -- 172.21.15.186:0/396379712 learned_addr learned my addr 172.21.15.186:0/396379712 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f1d9b640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99f40c57e0 0x7f99f40b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f2d9d640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99f4154bd0 0x7f99f41e5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f259c640 1 -- 172.21.15.186:0/396379712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99f4154bd0 msgr2=0x7f99f41e5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f259c640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99f4154bd0 0x7f99f41e5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f259c640 1 -- 172.21.15.186:0/396379712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99f40c57e0 msgr2=0x7f99f40b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f259c640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99f40c57e0 0x7f99f40b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f259c640 1 -- 172.21.15.186:0/396379712 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f41f0480 con 0x7f99f40c53c0 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f1d9b640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99f40c57e0 0x7f99f40b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f2d9d640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99f4154bd0 0x7f99f41e5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.696+0000 7f99f259c640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f40c53c0 0x7f99f415a460 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f99e4037040 tx=0x7f99e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.697+0000 7f99eaffd640 1 -- 172.21.15.186:0/396379712 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4039050 con 0x7f99f40c53c0 2024-03-28T15:09:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.697+0000 7f99eaffd640 1 -- 172.21.15.186:0/396379712 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99e403a040 con 0x7f99f40c53c0 2024-03-28T15:09:48.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.697+0000 7f99f8b63640 1 -- 172.21.15.186:0/396379712 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f41e3ca0 con 0x7f99f40c53c0 2024-03-28T15:09:48.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.697+0000 7f99eaffd640 1 -- 172.21.15.186:0/396379712 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4003bd0 con 0x7f99f40c53c0 2024-03-28T15:09:48.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.697+0000 7f99f8b63640 1 -- 172.21.15.186:0/396379712 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f41f0700 con 0x7f99f40c53c0 2024-03-28T15:09:48.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.698+0000 7f99f8b63640 1 -- 172.21.15.186:0/396379712 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99f4069500 con 0x7f99f40c53c0 2024-03-28T15:09:48.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.698+0000 7f99eaffd640 1 -- 172.21.15.186:0/396379712 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f99e4004030 con 0x7f99f40c53c0 2024-03-28T15:09:48.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.699+0000 7f99eaffd640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f99d406c560 0x7f99d406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:48.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.699+0000 7f99f1d9b640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f99d406c560 0x7f99d406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:48.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.699+0000 7f99eaffd640 1 -- 172.21.15.186:0/396379712 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7f99e40adeb0 con 0x7f99f40c53c0 2024-03-28T15:09:48.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.699+0000 7f99f1d9b640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f99d406c560 0x7f99d406ea20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f99f41e4530 tx=0x7f99dc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:48.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.703+0000 7f99eaffd640 1 -- 172.21.15.186:0/396379712 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f99f41f0480 con 0x7f99f40c53c0 2024-03-28T15:09:48.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.916+0000 7f99f8b63640 1 -- 172.21.15.186:0/396379712 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99f414ec50 con 0x7f99f40c53c0 2024-03-28T15:09:48.917 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.917+0000 7f99eaffd640 1 -- 172.21.15.186:0/396379712 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6848 (secure 0 0 0) 0x7f99e407c390 con 0x7f99f40c53c0 2024-03-28T15:09:48.918 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:09:48.918 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:49.855657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:active","state_seq":38,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5137,"qdb_cluster":[5137]},"id":1}]} 2024-03-28T15:09:48.918 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 29 2024-03-28T15:09:48.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 -- 172.21.15.186:0/396379712 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f99d406c560 msgr2=0x7f99d406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:48.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f99d406c560 0x7f99d406ea20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f99f41e4530 tx=0x7f99dc009040 comp rx=0 tx=0).stop 2024-03-28T15:09:48.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 -- 172.21.15.186:0/396379712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f40c53c0 msgr2=0x7f99f415a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:48.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f40c53c0 0x7f99f415a460 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f99e4037040 tx=0x7f99e4004a40 comp rx=0 tx=0).stop 2024-03-28T15:09:48.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 -- 172.21.15.186:0/396379712 shutdown_connections 2024-03-28T15:09:48.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f99d406c560 0x7f99d406ea20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:48.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99f4154bd0 0x7f99f41e5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:48.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99f40c57e0 0x7f99f40b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:48.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 --2- 172.21.15.186:0/396379712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f40c53c0 0x7f99f415a460 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:48.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 -- 172.21.15.186:0/396379712 >> 172.21.15.186:0/396379712 conn(0x7f99f40731f0 msgr2=0x7f99f40ce030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:48.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 -- 172.21.15.186:0/396379712 shutdown_connections 2024-03-28T15:09:48.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:48.919+0000 7f99f8b63640 1 -- 172.21.15.186:0/396379712 wait complete. 2024-03-28T15:09:48.930 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:09:50.931 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:09:51.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.044+0000 7f46652ed640 1 Processor -- start 2024-03-28T15:09:51.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.044+0000 7f46652ed640 1 -- start start 2024-03-28T15:09:51.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f46652ed640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46600cab20 0x7f466015bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:51.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f46652ed640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46600c8560 con 0x7f4660152b80 2024-03-28T15:09:51.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f46652ed640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46600c33f0 con 0x7f466015c180 2024-03-28T15:09:51.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f46652ed640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f46601590d0 con 0x7f46600cab20 2024-03-28T15:09:51.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f465effd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4660152b80 0x7f46600c4d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:50550/0 (socket says 172.21.15.186:50550) 2024-03-28T15:09:51.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f465effd640 1 -- 172.21.15.186:0/977626573 learned_addr learned my addr 172.21.15.186:0/977626573 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:09:51.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f465e7fc640 1 --2- 172.21.15.186:0/977626573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46600cab20 0x7f466015bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:51.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f465dffb640 1 -- 172.21.15.186:0/977626573 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3471875055 0 0) 0x7f46600c8560 con 0x7f4660152b80 2024-03-28T15:09:51.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f465dffb640 1 -- 172.21.15.186:0/977626573 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4648003520 con 0x7f4660152b80 2024-03-28T15:09:51.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f465dffb640 1 -- 172.21.15.186:0/977626573 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1826432138 0 0) 0x7f46600c33f0 con 0x7f466015c180 2024-03-28T15:09:51.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f465dffb640 1 -- 172.21.15.186:0/977626573 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46600c8560 con 0x7f466015c180 2024-03-28T15:09:51.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f465e7fc640 1 -- 172.21.15.186:0/977626573 >> v1:172.21.15.191:6789/0 conn(0x7f466015c180 legacy=0x7f466006d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:51.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f465e7fc640 1 -- 172.21.15.186:0/977626573 >> v1:172.21.15.186:6789/0 conn(0x7f4660152b80 legacy=0x7f46600c4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:51.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.045+0000 7f465e7fc640 1 -- 172.21.15.186:0/977626573 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46601587a0 con 0x7f46600cab20 2024-03-28T15:09:51.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.046+0000 7f465e7fc640 1 --2- 172.21.15.186:0/977626573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46600cab20 0x7f466015bc40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4660159250 tx=0x7f465402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b30e15827716c414 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:51.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.047+0000 7f465dffb640 1 -- 172.21.15.186:0/977626573 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4654034070 con 0x7f46600cab20 2024-03-28T15:09:51.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.047+0000 7f465dffb640 1 -- 172.21.15.186:0/977626573 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4654030880 con 0x7f46600cab20 2024-03-28T15:09:51.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.047+0000 7f465dffb640 1 -- 172.21.15.186:0/977626573 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4654030c50 con 0x7f46600cab20 2024-03-28T15:09:51.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.047+0000 7f46652ed640 1 -- 172.21.15.186:0/977626573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46600cab20 msgr2=0x7f466015bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:51.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.047+0000 7f46652ed640 1 --2- 172.21.15.186:0/977626573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46600cab20 0x7f466015bc40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4660159250 tx=0x7f465402b9c0 comp rx=0 tx=0).stop 2024-03-28T15:09:51.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.047+0000 7f46652ed640 1 -- 172.21.15.186:0/977626573 shutdown_connections 2024-03-28T15:09:51.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.047+0000 7f46652ed640 1 --2- 172.21.15.186:0/977626573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46600cab20 0x7f466015bc40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:51.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.047+0000 7f46652ed640 1 -- 172.21.15.186:0/977626573 >> 172.21.15.186:0/977626573 conn(0x7f4660067280 msgr2=0x7f4660067660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:51.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.047+0000 7f46652ed640 1 -- 172.21.15.186:0/977626573 shutdown_connections 2024-03-28T15:09:51.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.047+0000 7f46652ed640 1 -- 172.21.15.186:0/977626573 wait complete. 2024-03-28T15:09:51.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.048+0000 7f46652ed640 1 Processor -- start 2024-03-28T15:09:51.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f46652ed640 1 -- start start 2024-03-28T15:09:51.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f46652ed640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46600cab20 0x7f46601ef120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:51.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f46652ed640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4660152b80 0x7f46601ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:51.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f46652ed640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f466015c180 0x7f466015a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:51.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f46652ed640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4660159560 con 0x7f46600cab20 2024-03-28T15:09:51.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f46652ed640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4660158d60 con 0x7f4660152b80 2024-03-28T15:09:51.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f46652ed640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f46600c30b0 con 0x7f466015c180 2024-03-28T15:09:51.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f465effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46600cab20 0x7f46601ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f465effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46600cab20 0x7f46601ef120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:38308/0 (socket says 172.21.15.186:38308) 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f465effd640 1 -- 172.21.15.186:0/310223866 learned_addr learned my addr 172.21.15.186:0/310223866 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f465e7fc640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4660152b80 0x7f46601ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.049+0000 7f465f7fe640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f466015c180 0x7f466015a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.050+0000 7f465effd640 1 -- 172.21.15.186:0/310223866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f466015c180 msgr2=0x7f466015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.050+0000 7f465effd640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f466015c180 0x7f466015a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.050+0000 7f465effd640 1 -- 172.21.15.186:0/310223866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4660152b80 msgr2=0x7f46601ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.050+0000 7f465effd640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4660152b80 0x7f46601ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.050+0000 7f465effd640 1 -- 172.21.15.186:0/310223866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4660150470 con 0x7f46600cab20 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.050+0000 7f465effd640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46600cab20 0x7f46601ef120 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f465800c530 tx=0x7f465800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.050+0000 7f46477fe640 1 -- 172.21.15.186:0/310223866 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4658017070 con 0x7f46600cab20 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.050+0000 7f46477fe640 1 -- 172.21.15.186:0/310223866 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4658005d00 con 0x7f46600cab20 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.050+0000 7f46477fe640 1 -- 172.21.15.186:0/310223866 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46580072b0 con 0x7f46600cab20 2024-03-28T15:09:51.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.050+0000 7f46652ed640 1 -- 172.21.15.186:0/310223866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4660150190 con 0x7f46600cab20 2024-03-28T15:09:51.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.050+0000 7f46652ed640 1 -- 172.21.15.186:0/310223866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4660159b90 con 0x7f46600cab20 2024-03-28T15:09:51.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.051+0000 7f46652ed640 1 -- 172.21.15.186:0/310223866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4660069500 con 0x7f46600cab20 2024-03-28T15:09:51.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.052+0000 7f46477fe640 1 -- 172.21.15.186:0/310223866 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f465801a070 con 0x7f46600cab20 2024-03-28T15:09:51.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.052+0000 7f46477fe640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f464c06c560 0x7f464c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:51.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.052+0000 7f46477fe640 1 -- 172.21.15.186:0/310223866 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7f465808ab20 con 0x7f46600cab20 2024-03-28T15:09:51.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.052+0000 7f465e7fc640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f464c06c560 0x7f464c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:51.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.053+0000 7f465e7fc640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f464c06c560 0x7f464c06ea20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4654036000 tx=0x7f465403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:51.057 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.057+0000 7f46477fe640 1 -- 172.21.15.186:0/310223866 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4658058a70 con 0x7f46600cab20 2024-03-28T15:09:51.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.270+0000 7f46652ed640 1 -- 172.21.15.186:0/310223866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46600681f0 con 0x7f46600cab20 2024-03-28T15:09:51.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.271+0000 7f46477fe640 1 -- 172.21.15.186:0/310223866 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6848 (secure 0 0 0) 0x7f465805cd70 con 0x7f46600cab20 2024-03-28T15:09:51.272 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:09:51.272 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:49.855657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:active","state_seq":38,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5137,"qdb_cluster":[5137]},"id":1}]} 2024-03-28T15:09:51.272 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 29 2024-03-28T15:09:51.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 -- 172.21.15.186:0/310223866 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f464c06c560 msgr2=0x7f464c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:51.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f464c06c560 0x7f464c06ea20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4654036000 tx=0x7f465403b040 comp rx=0 tx=0).stop 2024-03-28T15:09:51.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 -- 172.21.15.186:0/310223866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46600cab20 msgr2=0x7f46601ef120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:51.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46600cab20 0x7f46601ef120 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f465800c530 tx=0x7f465800c9f0 comp rx=0 tx=0).stop 2024-03-28T15:09:51.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 -- 172.21.15.186:0/310223866 shutdown_connections 2024-03-28T15:09:51.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f464c06c560 0x7f464c06ea20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:51.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f466015c180 0x7f466015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:51.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4660152b80 0x7f46601ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:51.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 --2- 172.21.15.186:0/310223866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46600cab20 0x7f46601ef120 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:51.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 -- 172.21.15.186:0/310223866 >> 172.21.15.186:0/310223866 conn(0x7f4660067280 msgr2=0x7f46600cd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:51.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 -- 172.21.15.186:0/310223866 shutdown_connections 2024-03-28T15:09:51.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:51.273+0000 7f46652ed640 1 -- 172.21.15.186:0/310223866 wait complete. 2024-03-28T15:09:51.285 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:09:53.286 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:09:53.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.398+0000 7f36f4d93640 1 Processor -- start 2024-03-28T15:09:53.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.398+0000 7f36f4d93640 1 -- start start 2024-03-28T15:09:53.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.399+0000 7f36f4d93640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f36f0154bd0 0x7f36f015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:53.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.399+0000 7f36f4d93640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f36f00c57e0 0x7f36f00cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:53.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.399+0000 7f36f4d93640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36f00c6e00 con 0x7f36f00c53c0 2024-03-28T15:09:53.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.399+0000 7f36f4d93640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f36f00c5ca0 con 0x7f36f00c57e0 2024-03-28T15:09:53.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.399+0000 7f36f4d93640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f36f015b080 con 0x7f36f0154bd0 2024-03-28T15:09:53.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.399+0000 7f36eed9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f36f00c57e0 0x7f36f00cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:53.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.400+0000 7f36edd9b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f36f00c53c0 0x7f36f00c66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:50566/0 (socket says 172.21.15.186:50566) 2024-03-28T15:09:53.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.400+0000 7f36eed9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f36f00c57e0 0x7f36f00cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:38318/0 (socket says 172.21.15.186:38318) 2024-03-28T15:09:53.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.400+0000 7f36edd9b640 1 -- 172.21.15.186:0/1332905755 learned_addr learned my addr 172.21.15.186:0/1332905755 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:09:53.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.400+0000 7f36ee59c640 1 --2- 172.21.15.186:0/1332905755 >> v2:172.21.15.191:3300/0 conn(0x7f36f0154bd0 0x7f36f015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:53.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.400+0000 7f36eed9d640 1 -- 172.21.15.186:0/1332905755 >> v2:172.21.15.191:3300/0 conn(0x7f36f0154bd0 msgr2=0x7f36f015a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:09:53.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.400+0000 7f36eed9d640 1 --2- 172.21.15.186:0/1332905755 >> v2:172.21.15.191:3300/0 conn(0x7f36f0154bd0 0x7f36f015a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:53.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.400+0000 7f36eed9d640 1 -- 172.21.15.186:0/1332905755 >> v1:172.21.15.186:6789/0 conn(0x7f36f00c53c0 legacy=0x7f36f00c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:53.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.400+0000 7f36eed9d640 1 -- 172.21.15.186:0/1332905755 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f015b360 con 0x7f36f00c57e0 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36eed9d640 1 --2- 172.21.15.186:0/1332905755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36f00c57e0 0x7f36f00cb040 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f36f00d0b90 tx=0x7f36e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc69d31d1db6f4b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36ed59a640 1 -- 172.21.15.186:0/1332905755 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8034070 con 0x7f36f00c57e0 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36ed59a640 1 -- 172.21.15.186:0/1332905755 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36e8030610 con 0x7f36f00c57e0 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36ed59a640 1 -- 172.21.15.186:0/1332905755 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e80309e0 con 0x7f36f00c57e0 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36f4d93640 1 -- 172.21.15.186:0/1332905755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36f00c57e0 msgr2=0x7f36f00cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36f4d93640 1 --2- 172.21.15.186:0/1332905755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36f00c57e0 0x7f36f00cb040 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f36f00d0b90 tx=0x7f36e802b960 comp rx=0 tx=0).stop 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36f4d93640 1 -- 172.21.15.186:0/1332905755 shutdown_connections 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36f4d93640 1 --2- 172.21.15.186:0/1332905755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36f00c57e0 0x7f36f00cb040 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36f4d93640 1 --2- 172.21.15.186:0/1332905755 >> v2:172.21.15.191:3300/0 conn(0x7f36f0154bd0 0x7f36f015a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36f4d93640 1 -- 172.21.15.186:0/1332905755 >> 172.21.15.186:0/1332905755 conn(0x7f36f00731f0 msgr2=0x7f36f0073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36f4d93640 1 -- 172.21.15.186:0/1332905755 shutdown_connections 2024-03-28T15:09:53.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.401+0000 7f36f4d93640 1 -- 172.21.15.186:0/1332905755 wait complete. 2024-03-28T15:09:53.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.403+0000 7f36f4d93640 1 Processor -- start 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.403+0000 7f36f4d93640 1 -- start start 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.403+0000 7f36f4d93640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36f00c53c0 0x7f36f01f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.403+0000 7f36f4d93640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36f00c57e0 0x7f36f0153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.403+0000 7f36f4d93640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36f0154bd0 0x7f36f0143870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.403+0000 7f36f4d93640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36f00c61d0 con 0x7f36f00c53c0 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36f4d93640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f36f00c5ef0 con 0x7f36f0154bd0 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36f4d93640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f36f00cc390 con 0x7f36f00c57e0 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36ee59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36f00c53c0 0x7f36f01f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36ee59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36f00c53c0 0x7f36f01f3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:38320/0 (socket says 172.21.15.186:38320) 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36ee59c640 1 -- 172.21.15.186:0/1898191958 learned_addr learned my addr 172.21.15.186:0/1898191958 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36edd9b640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36f00c57e0 0x7f36f0153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36eed9d640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36f0154bd0 0x7f36f0143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:53.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36ee59c640 1 -- 172.21.15.186:0/1898191958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36f00c57e0 msgr2=0x7f36f0153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:53.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36ee59c640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36f00c57e0 0x7f36f0153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:53.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36ee59c640 1 -- 172.21.15.186:0/1898191958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36f0154bd0 msgr2=0x7f36f0143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:53.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36ee59c640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36f0154bd0 0x7f36f0143870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:53.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36ee59c640 1 -- 172.21.15.186:0/1898191958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f01f41d0 con 0x7f36f00c53c0 2024-03-28T15:09:53.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36ee59c640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36f00c53c0 0x7f36f01f3c90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f36e0007e70 tx=0x7f36e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:53.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36deffd640 1 -- 172.21.15.186:0/1898191958 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e0014620 con 0x7f36f00c53c0 2024-03-28T15:09:53.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36deffd640 1 -- 172.21.15.186:0/1898191958 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36e0019070 con 0x7f36f00c53c0 2024-03-28T15:09:53.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36f4d93640 1 -- 172.21.15.186:0/1898191958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f0147e60 con 0x7f36f00c53c0 2024-03-28T15:09:53.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36deffd640 1 -- 172.21.15.186:0/1898191958 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e0014a00 con 0x7f36f00c53c0 2024-03-28T15:09:53.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.404+0000 7f36f4d93640 1 -- 172.21.15.186:0/1898191958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f01483e0 con 0x7f36f00c53c0 2024-03-28T15:09:53.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.405+0000 7f36f4d93640 1 -- 172.21.15.186:0/1898191958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36f01e6770 con 0x7f36f00c53c0 2024-03-28T15:09:53.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.406+0000 7f36deffd640 1 -- 172.21.15.186:0/1898191958 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f36e0014ba0 con 0x7f36f00c53c0 2024-03-28T15:09:53.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.406+0000 7f36deffd640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f36d006c530 0x7f36d006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:53.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.406+0000 7f36deffd640 1 -- 172.21.15.186:0/1898191958 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7f36e008bfe0 con 0x7f36f00c53c0 2024-03-28T15:09:53.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.406+0000 7f36edd9b640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f36d006c530 0x7f36d006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:53.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.407+0000 7f36edd9b640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f36d006c530 0x7f36d006e9f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f36f01e6ff0 tx=0x7f36e4007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:53.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.411+0000 7f36deffd640 1 -- 172.21.15.186:0/1898191958 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f36e0059de0 con 0x7f36f00c53c0 2024-03-28T15:09:53.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.622+0000 7f36f4d93640 1 -- 172.21.15.186:0/1898191958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36f0069500 con 0x7f36f00c53c0 2024-03-28T15:09:53.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.622+0000 7f36deffd640 1 -- 172.21.15.186:0/1898191958 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6848 (secure 0 0 0) 0x7f36e005e0e0 con 0x7f36f00c53c0 2024-03-28T15:09:53.623 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:09:53.623 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:49.855657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:active","state_seq":38,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5137,"qdb_cluster":[5137]},"id":1}]} 2024-03-28T15:09:53.623 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 29 2024-03-28T15:09:53.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 -- 172.21.15.186:0/1898191958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f36d006c530 msgr2=0x7f36d006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:53.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f36d006c530 0x7f36d006e9f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f36f01e6ff0 tx=0x7f36e4007e80 comp rx=0 tx=0).stop 2024-03-28T15:09:53.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 -- 172.21.15.186:0/1898191958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36f00c53c0 msgr2=0x7f36f01f3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:53.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36f00c53c0 0x7f36f01f3c90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f36e0007e70 tx=0x7f36e000e4b0 comp rx=0 tx=0).stop 2024-03-28T15:09:53.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 -- 172.21.15.186:0/1898191958 shutdown_connections 2024-03-28T15:09:53.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f36d006c530 0x7f36d006e9f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:53.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36f0154bd0 0x7f36f0143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:53.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36f00c57e0 0x7f36f0153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:53.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 --2- 172.21.15.186:0/1898191958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36f00c53c0 0x7f36f01f3c90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:53.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 -- 172.21.15.186:0/1898191958 >> 172.21.15.186:0/1898191958 conn(0x7f36f00731f0 msgr2=0x7f36f014a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:53.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 -- 172.21.15.186:0/1898191958 shutdown_connections 2024-03-28T15:09:53.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:53.624+0000 7f36f4d93640 1 -- 172.21.15.186:0/1898191958 wait complete. 2024-03-28T15:09:53.636 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:09:55.637 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:09:55.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.749+0000 7f8fcc31e640 1 Processor -- start 2024-03-28T15:09:55.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.749+0000 7f8fcc31e640 1 -- start start 2024-03-28T15:09:55.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.750+0000 7f8fcc31e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8fc4151990 0x7f8fc40c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:55.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.750+0000 7f8fcc31e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8fc40b1e30 0x7f8fc40c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:55.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.750+0000 7f8fcc31e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc406caa0 con 0x7f8fc40c2d00 2024-03-28T15:09:55.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.750+0000 7f8fcc31e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8fc40c1da0 con 0x7f8fc40b1e30 2024-03-28T15:09:55.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.750+0000 7f8fcc31e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f8fc406cd80 con 0x7f8fc4151990 2024-03-28T15:09:55.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.750+0000 7f8fc98b9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8fc40b1e30 0x7f8fc40c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:55.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.750+0000 7f8fca8bb640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8fc40c2d00 0x7f8fc40712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:50582/0 (socket says 172.21.15.186:50582) 2024-03-28T15:09:55.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.750+0000 7f8fc98b9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8fc40b1e30 0x7f8fc40c2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:38336/0 (socket says 172.21.15.186:38336) 2024-03-28T15:09:55.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.750+0000 7f8fca8bb640 1 -- 172.21.15.186:0/4037658949 learned_addr learned my addr 172.21.15.186:0/4037658949 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:09:55.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.750+0000 7f8fca0ba640 1 --2- 172.21.15.186:0/4037658949 >> v2:172.21.15.191:3300/0 conn(0x7f8fc4151990 0x7f8fc40c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:55.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.751+0000 7f8fc98b9640 1 -- 172.21.15.186:0/4037658949 >> v2:172.21.15.191:3300/0 conn(0x7f8fc4151990 msgr2=0x7f8fc40c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:09:55.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.751+0000 7f8fc98b9640 1 --2- 172.21.15.186:0/4037658949 >> v2:172.21.15.191:3300/0 conn(0x7f8fc4151990 0x7f8fc40c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:55.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.751+0000 7f8fc98b9640 1 -- 172.21.15.186:0/4037658949 >> v1:172.21.15.186:6789/0 conn(0x7f8fc40c2d00 legacy=0x7f8fc40712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:55.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.751+0000 7f8fc98b9640 1 -- 172.21.15.186:0/4037658949 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc40c1340 con 0x7f8fc40b1e30 2024-03-28T15:09:55.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.751+0000 7f8fc90b8640 1 -- 172.21.15.186:0/4037658949 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1365292742 0 0) 0x7f8fc406caa0 con 0x7f8fc40c2d00 2024-03-28T15:09:55.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.751+0000 7f8fca0ba640 1 --2- 172.21.15.186:0/4037658949 >> v2:172.21.15.191:3300/0 conn(0x7f8fc4151990 0x7f8fc40c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:09:55.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.751+0000 7f8fc98b9640 1 --2- 172.21.15.186:0/4037658949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40b1e30 0x7f8fc40c2770 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8fc406fd90 tx=0x7f8fb402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3913130dc24469e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:55.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.752+0000 7f8fc90b8640 1 -- 172.21.15.186:0/4037658949 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4034070 con 0x7f8fc40b1e30 2024-03-28T15:09:55.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.752+0000 7f8fc90b8640 1 -- 172.21.15.186:0/4037658949 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fb4030610 con 0x7f8fc40b1e30 2024-03-28T15:09:55.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.752+0000 7f8fc90b8640 1 -- 172.21.15.186:0/4037658949 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb40309e0 con 0x7f8fc40b1e30 2024-03-28T15:09:55.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.752+0000 7f8fcc31e640 1 -- 172.21.15.186:0/4037658949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40b1e30 msgr2=0x7f8fc40c2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:55.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.752+0000 7f8fcc31e640 1 --2- 172.21.15.186:0/4037658949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40b1e30 0x7f8fc40c2770 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8fc406fd90 tx=0x7f8fb402b960 comp rx=0 tx=0).stop 2024-03-28T15:09:55.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.752+0000 7f8fcc31e640 1 -- 172.21.15.186:0/4037658949 shutdown_connections 2024-03-28T15:09:55.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.752+0000 7f8fcc31e640 1 --2- 172.21.15.186:0/4037658949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40b1e30 0x7f8fc40c2770 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:55.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.752+0000 7f8fcc31e640 1 --2- 172.21.15.186:0/4037658949 >> v2:172.21.15.191:3300/0 conn(0x7f8fc4151990 0x7f8fc40c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:55.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.752+0000 7f8fcc31e640 1 -- 172.21.15.186:0/4037658949 >> 172.21.15.186:0/4037658949 conn(0x7f8fc40c8a20 msgr2=0x7f8fc40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:55.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.752+0000 7f8fcc31e640 1 -- 172.21.15.186:0/4037658949 shutdown_connections 2024-03-28T15:09:55.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.752+0000 7f8fcc31e640 1 -- 172.21.15.186:0/4037658949 wait complete. 2024-03-28T15:09:55.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.753+0000 7f8fcc31e640 1 Processor -- start 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.754+0000 7f8fcc31e640 1 -- start start 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.754+0000 7f8fcc31e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40b1e30 0x7f8fc4142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.754+0000 7f8fcc31e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fc40c2d00 0x7f8fc41eda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.754+0000 7f8fcc31e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fc4151990 0x7f8fc4146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.754+0000 7f8fcc31e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc4073dd0 con 0x7f8fc40b1e30 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.754+0000 7f8fcc31e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc40c1900 con 0x7f8fc4151990 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.754+0000 7f8fcc31e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8fc4073f50 con 0x7f8fc40c2d00 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.754+0000 7f8fca0ba640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40b1e30 0x7f8fc4142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.754+0000 7f8fca0ba640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40b1e30 0x7f8fc4142710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:38348/0 (socket says 172.21.15.186:38348) 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.754+0000 7f8fca0ba640 1 -- 172.21.15.186:0/1061564997 learned_addr learned my addr 172.21.15.186:0/1061564997 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.754+0000 7f8fc98b9640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fc40c2d00 0x7f8fc41eda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:55.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fca8bb640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fc4151990 0x7f8fc4146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fca0ba640 1 -- 172.21.15.186:0/1061564997 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fc40c2d00 msgr2=0x7f8fc41eda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fca0ba640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fc40c2d00 0x7f8fc41eda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fca0ba640 1 -- 172.21.15.186:0/1061564997 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fc4151990 msgr2=0x7f8fc4146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fca0ba640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fc4151990 0x7f8fc4146fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fca0ba640 1 -- 172.21.15.186:0/1061564997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc4141180 con 0x7f8fc40b1e30 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fca0ba640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40b1e30 0x7f8fc4142710 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000d100 tx=0x7f8fc000d5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fbaffd640 1 -- 172.21.15.186:0/1061564997 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc001a070 con 0x7f8fc40b1e30 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fbaffd640 1 -- 172.21.15.186:0/1061564997 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc0003b90 con 0x7f8fc40b1e30 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fcc31e640 1 -- 172.21.15.186:0/1061564997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc414ef00 con 0x7f8fc40b1e30 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fbaffd640 1 -- 172.21.15.186:0/1061564997 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0004030 con 0x7f8fc40b1e30 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.755+0000 7f8fcc31e640 1 -- 172.21.15.186:0/1061564997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fc4141720 con 0x7f8fc40b1e30 2024-03-28T15:09:55.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.756+0000 7f8fcc31e640 1 -- 172.21.15.186:0/1061564997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fc4068a50 con 0x7f8fc40b1e30 2024-03-28T15:09:55.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.757+0000 7f8fbaffd640 1 -- 172.21.15.186:0/1061564997 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8fc001d070 con 0x7f8fc40b1e30 2024-03-28T15:09:55.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.757+0000 7f8fbaffd640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8fa806c530 0x7f8fa806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:55.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.757+0000 7f8fbaffd640 1 -- 172.21.15.186:0/1061564997 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6169+0+0 (secure 0 0 0) 0x7f8fc008bae0 con 0x7f8fc40b1e30 2024-03-28T15:09:55.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.757+0000 7f8fc98b9640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8fa806c530 0x7f8fa806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:55.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.758+0000 7f8fc98b9640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8fa806c530 0x7f8fa806e9f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4036000 tx=0x7f8fb403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:55.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.762+0000 7f8fbaffd640 1 -- 172.21.15.186:0/1061564997 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8fc4068a50 con 0x7f8fc40b1e30 2024-03-28T15:09:55.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.979+0000 7f8fcc31e640 1 -- 172.21.15.186:0/1061564997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fc4069500 con 0x7f8fc40b1e30 2024-03-28T15:09:55.980 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.979+0000 7f8fbaffd640 1 -- 172.21.15.186:0/1061564997 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6848 (secure 0 0 0) 0x7f8fc0010070 con 0x7f8fc40b1e30 2024-03-28T15:09:55.981 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:09:55.981 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:08:49.855657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5137":{"gid":5137,"name":"f","rank":0,"incarnation":25,"state":"up:active","state_seq":38,"addr":"172.21.15.191:6837/3037586083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3037586083},{"type":"v1","addr":"172.21.15.191:6837","nonce":3037586083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5137,"qdb_cluster":[5137]},"id":1}]} 2024-03-28T15:09:55.981 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 29 2024-03-28T15:09:55.982 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 -- 172.21.15.186:0/1061564997 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8fa806c530 msgr2=0x7f8fa806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:55.982 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8fa806c530 0x7f8fa806e9f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4036000 tx=0x7f8fb403b040 comp rx=0 tx=0).stop 2024-03-28T15:09:55.982 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 -- 172.21.15.186:0/1061564997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40b1e30 msgr2=0x7f8fc4142710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:55.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40b1e30 0x7f8fc4142710 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000d100 tx=0x7f8fc000d5c0 comp rx=0 tx=0).stop 2024-03-28T15:09:55.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 -- 172.21.15.186:0/1061564997 shutdown_connections 2024-03-28T15:09:55.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8fa806c530 0x7f8fa806e9f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:55.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fc4151990 0x7f8fc4146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:55.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fc40c2d00 0x7f8fc41eda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:55.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 --2- 172.21.15.186:0/1061564997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40b1e30 0x7f8fc4142710 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:55.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 -- 172.21.15.186:0/1061564997 >> 172.21.15.186:0/1061564997 conn(0x7f8fc40c8a20 msgr2=0x7f8fc406d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:55.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 -- 172.21.15.186:0/1061564997 shutdown_connections 2024-03-28T15:09:55.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:55.981+0000 7f8fcc31e640 1 -- 172.21.15.186:0/1061564997 wait complete. 2024-03-28T15:09:55.993 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-28T15:09:57.567 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:09:57.570 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:09:57.994 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:09:58.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.103+0000 7f6f7bf2c640 1 Processor -- start 2024-03-28T15:09:58.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.103+0000 7f6f7bf2c640 1 -- start start 2024-03-28T15:09:58.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f7bf2c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6f74152810 0x7f6f74158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:58.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f7bf2c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f74071340 0x7f6f7406c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:58.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f7bf2c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f74072df0 con 0x7f6f7406ce60 2024-03-28T15:09:58.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f7bf2c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f6f74159500 con 0x7f6f74152810 2024-03-28T15:09:58.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f7bf2c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f6f740c8660 con 0x7f6f74071340 2024-03-28T15:09:58.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f7a4c9640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6f7406ce60 0x7f6f740c6cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:50586/0 (socket says 172.21.15.186:50586) 2024-03-28T15:09:58.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f7a4c9640 1 -- 172.21.15.186:0/3513294724 learned_addr learned my addr 172.21.15.186:0/3513294724 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:09:58.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f794c7640 1 --2- 172.21.15.186:0/3513294724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f74071340 0x7f6f7406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:58.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f79cc8640 1 --2- 172.21.15.186:0/3513294724 >> v2:172.21.15.191:3300/0 conn(0x7f6f74152810 0x7f6f74158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:58.105 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f78cc6640 1 -- 172.21.15.186:0/3513294724 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 538744696 0 0) 0x7f6f74072df0 con 0x7f6f7406ce60 2024-03-28T15:09:58.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f78cc6640 1 -- 172.21.15.186:0/3513294724 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f60003520 con 0x7f6f7406ce60 2024-03-28T15:09:58.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f794c7640 1 -- 172.21.15.186:0/3513294724 >> v2:172.21.15.191:3300/0 conn(0x7f6f74152810 msgr2=0x7f6f74158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:09:58.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f794c7640 1 --2- 172.21.15.186:0/3513294724 >> v2:172.21.15.191:3300/0 conn(0x7f6f74152810 0x7f6f74158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f78cc6640 1 -- 172.21.15.186:0/3513294724 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1782626065 0 0) 0x7f6f60003520 con 0x7f6f7406ce60 2024-03-28T15:09:58.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f794c7640 1 -- 172.21.15.186:0/3513294724 >> v1:172.21.15.186:6789/0 conn(0x7f6f7406ce60 legacy=0x7f6f740c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.104+0000 7f6f794c7640 1 -- 172.21.15.186:0/3513294724 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f74071d50 con 0x7f6f74071340 2024-03-28T15:09:58.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.105+0000 7f6f794c7640 1 --2- 172.21.15.186:0/3513294724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f74071340 0x7f6f7406c920 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6f640097a0 tx=0x7f6f6402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e1a50268aa03557 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:58.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.105+0000 7f6f78cc6640 1 -- 172.21.15.186:0/3513294724 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64037070 con 0x7f6f74071340 2024-03-28T15:09:58.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.105+0000 7f6f78cc6640 1 -- 172.21.15.186:0/3513294724 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f64030510 con 0x7f6f74071340 2024-03-28T15:09:58.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.105+0000 7f6f78cc6640 1 -- 172.21.15.186:0/3513294724 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c0019f0 con 0x7f6f74071340 2024-03-28T15:09:58.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.106+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3513294724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f74071340 msgr2=0x7f6f7406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.106+0000 7f6f7bf2c640 1 --2- 172.21.15.186:0/3513294724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f74071340 0x7f6f7406c920 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6f640097a0 tx=0x7f6f6402b790 comp rx=0 tx=0).stop 2024-03-28T15:09:58.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.106+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3513294724 shutdown_connections 2024-03-28T15:09:58.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.106+0000 7f6f7bf2c640 1 --2- 172.21.15.186:0/3513294724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f74071340 0x7f6f7406c920 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.106+0000 7f6f7bf2c640 1 --2- 172.21.15.186:0/3513294724 >> v2:172.21.15.191:3300/0 conn(0x7f6f74152810 0x7f6f74158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.106+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3513294724 >> 172.21.15.186:0/3513294724 conn(0x7f6f740c8a20 msgr2=0x7f6f740c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:58.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.106+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3513294724 shutdown_connections 2024-03-28T15:09:58.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.106+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3513294724 wait complete. 2024-03-28T15:09:58.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.107+0000 7f6f7bf2c640 1 Processor -- start 2024-03-28T15:09:58.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.107+0000 7f6f7bf2c640 1 -- start start 2024-03-28T15:09:58.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f7bf2c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f7406ce60 0x7f6f740ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f7bf2c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f74071340 0x7f6f741f0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f7bf2c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f74152810 0x7f6f7414cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f7bf2c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6f740725f0 con 0x7f6f74071340 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f7bf2c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6f74072310 con 0x7f6f74152810 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f7bf2c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f6f740728d0 con 0x7f6f7406ce60 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f794c7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f74071340 0x7f6f741f0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f794c7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f74071340 0x7f6f741f0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:38362/0 (socket says 172.21.15.186:38362) 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f794c7640 1 -- 172.21.15.186:0/3398332306 learned_addr learned my addr 172.21.15.186:0/3398332306 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f79cc8640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f7406ce60 0x7f6f740ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f7a4c9640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f74152810 0x7f6f7414cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f794c7640 1 -- 172.21.15.186:0/3398332306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f7406ce60 msgr2=0x7f6f740ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f794c7640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f7406ce60 0x7f6f740ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f794c7640 1 -- 172.21.15.186:0/3398332306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f74152810 msgr2=0x7f6f7414cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f794c7640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f74152810 0x7f6f7414cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f794c7640 1 -- 172.21.15.186:0/3398332306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f741f1310 con 0x7f6f74071340 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f7a4c9640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f74152810 0x7f6f7414cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f794c7640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f74071340 0x7f6f741f0dd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6f64000c00 tx=0x7f6f640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.108+0000 7f6f6a7fc640 1 -- 172.21.15.186:0/3398332306 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64041020 con 0x7f6f74071340 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.109+0000 7f6f6a7fc640 1 -- 172.21.15.186:0/3398332306 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f6403c890 con 0x7f6f74071340 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.109+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3398332306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f74159150 con 0x7f6f74071340 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.109+0000 7f6f6a7fc640 1 -- 172.21.15.186:0/3398332306 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64037040 con 0x7f6f74071340 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.109+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3398332306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f740af310 con 0x7f6f74071340 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.109+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3398332306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f74069500 con 0x7f6f74071340 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.110+0000 7f6f6a7fc640 1 -- 172.21.15.186:0/3398332306 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6f64044020 con 0x7f6f74071340 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.110+0000 7f6f6a7fc640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6f5806c560 0x7f6f5806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.110+0000 7f6f6a7fc640 1 -- 172.21.15.186:0/3398332306 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f6f640ad360 con 0x7f6f74071340 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.110+0000 7f6f79cc8640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6f5806c560 0x7f6f5806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:58.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.111+0000 7f6f79cc8640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6f5806c560 0x7f6f5806ea20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009ad0 tx=0x7f6f70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:58.115 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.115+0000 7f6f6a7fc640 1 -- 172.21.15.186:0/3398332306 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f6f6404b030 con 0x7f6f74071340 2024-03-28T15:09:58.329 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.328+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3398332306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f740c77f0 con 0x7f6f74071340 2024-03-28T15:09:58.329 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.328+0000 7f6f6a7fc640 1 -- 172.21.15.186:0/3398332306 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6012 (secure 0 0 0) 0x7f6f6407b6d0 con 0x7f6f74071340 2024-03-28T15:09:58.330 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:09:58.330 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:09:56.237830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:09:58.331 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 30 2024-03-28T15:09:58.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3398332306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6f5806c560 msgr2=0x7f6f5806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6f5806c560 0x7f6f5806ea20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009ad0 tx=0x7f6f70008040 comp rx=0 tx=0).stop 2024-03-28T15:09:58.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3398332306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f74071340 msgr2=0x7f6f741f0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f74071340 0x7f6f741f0dd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6f64000c00 tx=0x7f6f640046c0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3398332306 shutdown_connections 2024-03-28T15:09:58.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f6f5806c560 0x7f6f5806ea20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f74152810 0x7f6f7414cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f74071340 0x7f6f741f0dd0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 --2- 172.21.15.186:0/3398332306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f7406ce60 0x7f6f740ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3398332306 >> 172.21.15.186:0/3398332306 conn(0x7f6f740c8a20 msgr2=0x7f6f7406e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:58.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3398332306 shutdown_connections 2024-03-28T15:09:58.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.330+0000 7f6f7bf2c640 1 -- 172.21.15.186:0/3398332306 wait complete. 2024-03-28T15:09:58.343 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-03-28T15:09:58.343 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-28T15:09:58.344 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:09:58.444 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.444+0000 7f8d54040640 1 Processor -- start 2024-03-28T15:09:58.445 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.444+0000 7f8d54040640 1 -- start start 2024-03-28T15:09:58.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.445+0000 7f8d54040640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8d4c0c53c0 0x7f8d4c0c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:58.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.445+0000 7f8d54040640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d4c15b080 con 0x7f8d4c154bd0 2024-03-28T15:09:58.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.445+0000 7f8d54040640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d4c15b360 con 0x7f8d4c0c57e0 2024-03-28T15:09:58.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.445+0000 7f8d54040640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8d4c0c5ca0 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.445+0000 7f8d515db640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8d4c0c53c0 0x7f8d4c0c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:58.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.445+0000 7f8d51ddc640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8d4c154bd0 0x7f8d4c15a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:50600/0 (socket says 172.21.15.186:50600) 2024-03-28T15:09:58.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.445+0000 7f8d515db640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8d4c0c53c0 0x7f8d4c0c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:38368/0 (socket says 172.21.15.186:38368) 2024-03-28T15:09:58.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.445+0000 7f8d51ddc640 1 -- 172.21.15.186:0/1426048537 learned_addr learned my addr 172.21.15.186:0/1426048537 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:09:58.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.446+0000 7f8d515db640 1 -- 172.21.15.186:0/1426048537 >> v1:172.21.15.191:6789/0 conn(0x7f8d4c0c57e0 legacy=0x7f8d4c0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:09:58.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.446+0000 7f8d515db640 1 -- 172.21.15.186:0/1426048537 >> v1:172.21.15.186:6789/0 conn(0x7f8d4c154bd0 legacy=0x7f8d4c15a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.446+0000 7f8d515db640 1 -- 172.21.15.186:0/1426048537 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d4c15b640 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.446+0000 7f8d515db640 1 --2- 172.21.15.186:0/1426048537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4c0c53c0 0x7f8d4c0c66f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0097a0 tx=0x7f8d3c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e7b07f464afadf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:58.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.446+0000 7f8d50dda640 1 -- 172.21.15.186:0/1426048537 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c034070 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.446+0000 7f8d50dda640 1 -- 172.21.15.186:0/1426048537 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d3c030610 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.446+0000 7f8d50dda640 1 -- 172.21.15.186:0/1426048537 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c0309e0 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.447+0000 7f8d54040640 1 -- 172.21.15.186:0/1426048537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4c0c53c0 msgr2=0x7f8d4c0c66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.447+0000 7f8d54040640 1 --2- 172.21.15.186:0/1426048537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4c0c53c0 0x7f8d4c0c66f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0097a0 tx=0x7f8d3c02b960 comp rx=0 tx=0).stop 2024-03-28T15:09:58.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.447+0000 7f8d54040640 1 -- 172.21.15.186:0/1426048537 shutdown_connections 2024-03-28T15:09:58.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.447+0000 7f8d54040640 1 --2- 172.21.15.186:0/1426048537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4c0c53c0 0x7f8d4c0c66f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.447+0000 7f8d54040640 1 -- 172.21.15.186:0/1426048537 >> 172.21.15.186:0/1426048537 conn(0x7f8d4c0731f0 msgr2=0x7f8d4c073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:58.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.447+0000 7f8d54040640 1 -- 172.21.15.186:0/1426048537 shutdown_connections 2024-03-28T15:09:58.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.447+0000 7f8d54040640 1 -- 172.21.15.186:0/1426048537 wait complete. 2024-03-28T15:09:58.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.448+0000 7f8d54040640 1 Processor -- start 2024-03-28T15:09:58.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.448+0000 7f8d54040640 1 -- start start 2024-03-28T15:09:58.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d54040640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4c0c53c0 0x7f8d4c1561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:58.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d54040640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d4c0c57e0 0x7f8d4c157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:58.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d54040640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d4c154bd0 0x7f8d4c1589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:58.451 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d54040640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4c0cc390 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d54040640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4c0c61d0 con 0x7f8d4c0c57e0 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d54040640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8d4c0cb750 con 0x7f8d4c154bd0 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d51ddc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4c0c53c0 0x7f8d4c1561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d51ddc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4c0c53c0 0x7f8d4c1561e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:38384/0 (socket says 172.21.15.186:38384) 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d51ddc640 1 -- 172.21.15.186:0/2741374896 learned_addr learned my addr 172.21.15.186:0/2741374896 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d525dd640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d4c154bd0 0x7f8d4c1589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d515db640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d4c0c57e0 0x7f8d4c157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d51ddc640 1 -- 172.21.15.186:0/2741374896 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d4c154bd0 msgr2=0x7f8d4c1589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d51ddc640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d4c154bd0 0x7f8d4c1589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d51ddc640 1 -- 172.21.15.186:0/2741374896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d4c0c57e0 msgr2=0x7f8d4c157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d51ddc640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d4c0c57e0 0x7f8d4c157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.449+0000 7f8d51ddc640 1 -- 172.21.15.186:0/2741374896 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d4c1593e0 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.450+0000 7f8d515db640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d4c0c57e0 0x7f8d4c157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.450+0000 7f8d51ddc640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4c0c53c0 0x7f8d4c1561e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800ca70 tx=0x7f8d4800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:58.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.450+0000 7f8d427fc640 1 -- 172.21.15.186:0/2741374896 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d48017070 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.450+0000 7f8d427fc640 1 -- 172.21.15.186:0/2741374896 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d4800ab70 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.450+0000 7f8d54040640 1 -- 172.21.15.186:0/2741374896 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d4c159650 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.450+0000 7f8d427fc640 1 -- 172.21.15.186:0/2741374896 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d48012400 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.450+0000 7f8d54040640 1 -- 172.21.15.186:0/2741374896 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d4c15a950 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.451+0000 7f8d54040640 1 -- 172.21.15.186:0/2741374896 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d4c069500 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.451+0000 7f8d427fc640 1 -- 172.21.15.186:0/2741374896 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8d4801a070 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.452+0000 7f8d427fc640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8d3006c530 0x7f8d3006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:09:58.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.452+0000 7f8d427fc640 1 -- 172.21.15.186:0/2741374896 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f8d4808b640 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.452+0000 7f8d515db640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8d3006c530 0x7f8d3006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:09:58.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.452+0000 7f8d515db640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8d3006c530 0x7f8d3006e9f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c036000 tx=0x7f8d3c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:09:58.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.456+0000 7f8d427fc640 1 -- 172.21.15.186:0/2741374896 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f8d48059a40 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.668+0000 7f8d54040640 1 -- 172.21.15.186:0/2741374896 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d4c0cbb60 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.669+0000 7f8d427fc640 1 -- 172.21.15.186:0/2741374896 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6012 (secure 0 0 0) 0x7f8d48059340 con 0x7f8d4c0c53c0 2024-03-28T15:09:58.669 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:09:58.669 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:09:56.237830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:09:58.670 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 30 2024-03-28T15:09:58.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 -- 172.21.15.186:0/2741374896 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8d3006c530 msgr2=0x7f8d3006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8d3006c530 0x7f8d3006e9f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c036000 tx=0x7f8d3c03b040 comp rx=0 tx=0).stop 2024-03-28T15:09:58.671 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 -- 172.21.15.186:0/2741374896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4c0c53c0 msgr2=0x7f8d4c1561e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:09:58.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4c0c53c0 0x7f8d4c1561e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800ca70 tx=0x7f8d4800cf30 comp rx=0 tx=0).stop 2024-03-28T15:09:58.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 -- 172.21.15.186:0/2741374896 shutdown_connections 2024-03-28T15:09:58.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f8d3006c530 0x7f8d3006e9f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d4c154bd0 0x7f8d4c1589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d4c0c57e0 0x7f8d4c157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 --2- 172.21.15.186:0/2741374896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d4c0c53c0 0x7f8d4c1561e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:09:58.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 -- 172.21.15.186:0/2741374896 >> 172.21.15.186:0/2741374896 conn(0x7f8d4c0731f0 msgr2=0x7f8d4c0b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:09:58.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 -- 172.21.15.186:0/2741374896 shutdown_connections 2024-03-28T15:09:58.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:09:58.671+0000 7f8d54040640 1 -- 172.21.15.186:0/2741374896 wait complete. 2024-03-28T15:09:58.683 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 30, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T15:01:31.425299+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5278": { "addr": "172.21.15.191:6834/3646031339", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 3646031339, "type": "v2" }, { "addr": "172.21.15.191:6834", "nonce": 3646031339, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5278, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T15:09:56.237830+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5278 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/1278716016", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6833", "nonce": 1278716016, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1278716016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6835/2068761001", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2068761001, "type": "v2" }, { "addr": "172.21.15.186:6835", "nonce": 2068761001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2344737463", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6837", "nonce": 2344737463, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2344737463, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4284, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/1548903381", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 1548903381, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 1548903381, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T15:10:00.684 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:00.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.798+0000 7f3e81d3e640 1 Processor -- start 2024-03-28T15:10:00.799 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.798+0000 7f3e81d3e640 1 -- start start 2024-03-28T15:10:00.801 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.799+0000 7f3e81d3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e7c0b1e30 0x7f3e7c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:00.801 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.799+0000 7f3e81d3e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3e7c0c2d00 0x7f3e7c0717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.799+0000 7f3e81d3e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e7c06caa0 con 0x7f3e7c151990 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.799+0000 7f3e81d3e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f3e7c0c1da0 con 0x7f3e7c0c2d00 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.799+0000 7f3e81d3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7c06cd80 con 0x7f3e7c0b1e30 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.799+0000 7f3e7b7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f3e7c151990 0x7f3e7c0c2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:46806/0 (socket says 172.21.15.186:46806) 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.799+0000 7f3e7b7fe640 1 -- 172.21.15.186:0/3195318690 learned_addr learned my addr 172.21.15.186:0/3195318690 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7affd640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e7c0b1e30 0x7f3e7c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7bfff640 1 --2- 172.21.15.186:0/3195318690 >> v2:172.21.15.191:3300/0 conn(0x7f3e7c0c2d00 0x7f3e7c0717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2599842411 0 0) 0x7f3e7c06caa0 con 0x7f3e7c151990 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e64003520 con 0x7f3e7c151990 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1503325346 0 0) 0x7f3e64003520 con 0x7f3e7c151990 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e7c0b1e30 msgr2=0x7f3e7c0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e7c0b1e30 0x7f3e7c0712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 >> v2:172.21.15.191:3300/0 conn(0x7f3e7c0c2d00 msgr2=0x7f3e7c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:00.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 --2- 172.21.15.186:0/3195318690 >> v2:172.21.15.191:3300/0 conn(0x7f3e7c0c2d00 0x7f3e7c0717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c0c1340 con 0x7f3e7c151990 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f3e6c0019f0 con 0x7f3e7c151990 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7affd640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e7c0b1e30 0x7f3e7c0712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 >> v1:172.21.15.186:6789/0 conn(0x7f3e7c151990 legacy=0x7f3e7c0c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e640039b0 0x7f3e64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e7c0b1e30 0x7f3e64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e64004d40 0x7f3e640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c0c1da0 con 0x7f3e64004d40 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c06cd80 con 0x7f3e640039b0 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3e6400b9c0 con 0x7f3e7c0b1e30 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7bfff640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e64004d40 0x7f3e640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7affd640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e7c0b1e30 0x7f3e64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7b7fe640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e640039b0 0x7f3e64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:00.803 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7bfff640 1 -- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e7c0b1e30 msgr2=0x7f3e64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:00.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7bfff640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e7c0b1e30 0x7f3e64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:00.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7bfff640 1 -- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e640039b0 msgr2=0x7f3e64003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:00.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7bfff640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e640039b0 0x7f3e64003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:00.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.800+0000 7f3e7bfff640 1 -- 172.21.15.186:0/3195318690 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e64003520 con 0x7f3e64004d40 2024-03-28T15:10:00.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.801+0000 7f3e7bfff640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e64004d40 0x7f3e640051c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009ac0 tx=0x7f3e64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e252e3a5d8b20f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:00.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.801+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74039070 con 0x7f3e64004d40 2024-03-28T15:10:00.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.801+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e7402c710 con 0x7f3e64004d40 2024-03-28T15:10:00.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.801+0000 7f3e7a7fc640 1 -- 172.21.15.186:0/3195318690 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74009690 con 0x7f3e64004d40 2024-03-28T15:10:00.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.802+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3195318690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e64004d40 msgr2=0x7f3e640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:00.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.802+0000 7f3e81d3e640 1 --2- 172.21.15.186:0/3195318690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e64004d40 0x7f3e640051c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009ac0 tx=0x7f3e64005830 comp rx=0 tx=0).stop 2024-03-28T15:10:00.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.802+0000 7f3e7bfff640 1 -- 172.21.15.186:0/3195318690 reap_dead start 2024-03-28T15:10:00.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.802+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3195318690 shutdown_connections 2024-03-28T15:10:00.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.802+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3195318690 >> 172.21.15.186:0/3195318690 conn(0x7f3e7c0c8a20 msgr2=0x7f3e7c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:00.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.802+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3195318690 shutdown_connections 2024-03-28T15:10:00.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.802+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3195318690 wait complete. 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.803+0000 7f3e81d3e640 1 Processor -- start 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.803+0000 7f3e81d3e640 1 -- start start 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e81d3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e64004d40 0x7f3e7c0cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e81d3e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e7c1359b0 0x7f3e7c13aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e81d3e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e7c13d850 0x7f3e7c13b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e81d3e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c071d30 con 0x7f3e7c1359b0 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e81d3e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c0c1900 con 0x7f3e7c13d850 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e81d3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7c0c2940 con 0x7f3e64004d40 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e7c1359b0 0x7f3e7c13aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e7c1359b0 0x7f3e7c13aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41642/0 (socket says 172.21.15.186:41642) 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7affd640 1 -- 172.21.15.186:0/3687676699 learned_addr learned my addr 172.21.15.186:0/3687676699 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7b7fe640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e64004d40 0x7f3e7c0cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7bfff640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e7c13d850 0x7f3e7c13b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:00.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7affd640 1 -- 172.21.15.186:0/3687676699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e64004d40 msgr2=0x7f3e7c0cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7affd640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e64004d40 0x7f3e7c0cb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7affd640 1 -- 172.21.15.186:0/3687676699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e7c13d850 msgr2=0x7f3e7c13b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7affd640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e7c13d850 0x7f3e7c13b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7affd640 1 -- 172.21.15.186:0/3687676699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c13e250 con 0x7f3e7c1359b0 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7bfff640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e7c13d850 0x7f3e7c13b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.804+0000 7f3e7affd640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e7c1359b0 0x7f3e7c13aba0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3e70011990 tx=0x7f3e70011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.805+0000 7f3e63fff640 1 -- 172.21.15.186:0/3687676699 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7000fad0 con 0x7f3e7c1359b0 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.805+0000 7f3e63fff640 1 -- 172.21.15.186:0/3687676699 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e7000fc70 con 0x7f3e7c1359b0 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.805+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3687676699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e7c139de0 con 0x7f3e7c1359b0 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.805+0000 7f3e63fff640 1 -- 172.21.15.186:0/3687676699 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e70014440 con 0x7f3e7c1359b0 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.805+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3687676699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e7c13a3a0 con 0x7f3e7c1359b0 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.806+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3687676699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e7c069500 con 0x7f3e7c1359b0 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.806+0000 7f3e63fff640 1 -- 172.21.15.186:0/3687676699 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3e70012040 con 0x7f3e7c1359b0 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.807+0000 7f3e63fff640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3e6806c560 0x7f3e6806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:00.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.807+0000 7f3e63fff640 1 -- 172.21.15.186:0/3687676699 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f3e7008bf10 con 0x7f3e7c1359b0 2024-03-28T15:10:00.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.807+0000 7f3e7b7fe640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3e6806c560 0x7f3e6806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:00.810 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.807+0000 7f3e7b7fe640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3e6806c560 0x7f3e6806ea20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c004d00 tx=0x7f3e6c006040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:00.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:00.811+0000 7f3e63fff640 1 -- 172.21.15.186:0/3687676699 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3e70019030 con 0x7f3e7c1359b0 2024-03-28T15:10:01.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.027+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3687676699 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e7c0c7430 con 0x7f3e7c1359b0 2024-03-28T15:10:01.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.027+0000 7f3e63fff640 1 -- 172.21.15.186:0/3687676699 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6012 (secure 0 0 0) 0x7f3e70059cc0 con 0x7f3e7c1359b0 2024-03-28T15:10:01.028 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:01.028 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:09:56.237830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:10:01.028 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 30 2024-03-28T15:10:01.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.029+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3687676699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3e6806c560 msgr2=0x7f3e6806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:01.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.029+0000 7f3e81d3e640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3e6806c560 0x7f3e6806ea20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c004d00 tx=0x7f3e6c006040 comp rx=0 tx=0).stop 2024-03-28T15:10:01.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.029+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3687676699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e7c1359b0 msgr2=0x7f3e7c13aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:01.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.029+0000 7f3e81d3e640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e7c1359b0 0x7f3e7c13aba0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3e70011990 tx=0x7f3e70011e50 comp rx=0 tx=0).stop 2024-03-28T15:10:01.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.029+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3687676699 shutdown_connections 2024-03-28T15:10:01.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.029+0000 7f3e81d3e640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3e6806c560 0x7f3e6806ea20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:01.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.029+0000 7f3e81d3e640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e7c13d850 0x7f3e7c13b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:01.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.029+0000 7f3e81d3e640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e7c1359b0 0x7f3e7c13aba0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:01.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.029+0000 7f3e81d3e640 1 --2- 172.21.15.186:0/3687676699 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e64004d40 0x7f3e7c0cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:01.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.029+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3687676699 >> 172.21.15.186:0/3687676699 conn(0x7f3e7c0c8a20 msgr2=0x7f3e7c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:01.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.030+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3687676699 shutdown_connections 2024-03-28T15:10:01.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:01.030+0000 7f3e81d3e640 1 -- 172.21.15.186:0/3687676699 wait complete. 2024-03-28T15:10:01.041 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:10:03.042 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:03.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.154+0000 7f68dbd77640 1 Processor -- start 2024-03-28T15:10:03.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.154+0000 7f68dbd77640 1 -- start start 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68dbd77640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f68d40b1e30 0x7f68d40712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68dbd77640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d40c2d00 0x7f68d40717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68dbd77640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68d406caa0 con 0x7f68d4151990 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68dbd77640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f68d40c1da0 con 0x7f68d40b1e30 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68dbd77640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f68d406cd80 con 0x7f68d40c2d00 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68d9312640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f68d40b1e30 0x7f68d40712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68d9312640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f68d40b1e30 0x7f68d40712b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41652/0 (socket says 172.21.15.186:41652) 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68d9312640 1 -- 172.21.15.186:0/3430899088 learned_addr learned my addr 172.21.15.186:0/3430899088 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68da314640 1 --2- 172.21.15.186:0/3430899088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d40c2d00 0x7f68d40717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68d9312640 1 -- 172.21.15.186:0/3430899088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d40c2d00 msgr2=0x7f68d40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68d9312640 1 --2- 172.21.15.186:0/3430899088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d40c2d00 0x7f68d40717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:03.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68d9312640 1 -- 172.21.15.186:0/3430899088 >> v1:172.21.15.191:6789/0 conn(0x7f68d4151990 legacy=0x7f68d40c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:03.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68d9312640 1 -- 172.21.15.186:0/3430899088 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d40c1340 con 0x7f68d40b1e30 2024-03-28T15:10:03.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.155+0000 7f68d8b11640 1 -- 172.21.15.186:0/3430899088 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3814733628 0 0) 0x7f68d406caa0 con 0x7f68d4151990 2024-03-28T15:10:03.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.156+0000 7f68d9312640 1 --2- 172.21.15.186:0/3430899088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d40b1e30 0x7f68d40712b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009870 tx=0x7f68c402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4fcf4d922950ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:03.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.156+0000 7f68d8b11640 1 -- 172.21.15.186:0/3430899088 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c402f840 con 0x7f68d40b1e30 2024-03-28T15:10:03.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.156+0000 7f68d8b11640 1 -- 172.21.15.186:0/3430899088 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c402f9e0 con 0x7f68d40b1e30 2024-03-28T15:10:03.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.156+0000 7f68d8b11640 1 -- 172.21.15.186:0/3430899088 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c402fdb0 con 0x7f68d40b1e30 2024-03-28T15:10:03.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.157+0000 7f68dbd77640 1 -- 172.21.15.186:0/3430899088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d40b1e30 msgr2=0x7f68d40712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:03.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.157+0000 7f68dbd77640 1 --2- 172.21.15.186:0/3430899088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d40b1e30 0x7f68d40712b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009870 tx=0x7f68c402ba20 comp rx=0 tx=0).stop 2024-03-28T15:10:03.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.157+0000 7f68dbd77640 1 -- 172.21.15.186:0/3430899088 shutdown_connections 2024-03-28T15:10:03.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.157+0000 7f68dbd77640 1 --2- 172.21.15.186:0/3430899088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d40c2d00 0x7f68d40717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:03.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.157+0000 7f68dbd77640 1 --2- 172.21.15.186:0/3430899088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d40b1e30 0x7f68d40712b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:03.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.157+0000 7f68dbd77640 1 -- 172.21.15.186:0/3430899088 >> 172.21.15.186:0/3430899088 conn(0x7f68d40c8a20 msgr2=0x7f68d40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:03.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.157+0000 7f68dbd77640 1 -- 172.21.15.186:0/3430899088 shutdown_connections 2024-03-28T15:10:03.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.157+0000 7f68dbd77640 1 -- 172.21.15.186:0/3430899088 wait complete. 2024-03-28T15:10:03.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.158+0000 7f68dbd77640 1 Processor -- start 2024-03-28T15:10:03.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.158+0000 7f68dbd77640 1 -- start start 2024-03-28T15:10:03.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68dbd77640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d40b1e30 0x7f68d414d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:03.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68dbd77640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d40c2d00 0x7f68d40c5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:03.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68dbd77640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d4151990 0x7f68d40c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:03.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68dbd77640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f68d4071d30 con 0x7f68d40b1e30 2024-03-28T15:10:03.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68dbd77640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f68d40c1900 con 0x7f68d40c2d00 2024-03-28T15:10:03.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68dbd77640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f68d40c2940 con 0x7f68d4151990 2024-03-28T15:10:03.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68d9b13640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d40b1e30 0x7f68d414d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:03.161 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68d9312640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d40c2d00 0x7f68d40c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68d9b13640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d40b1e30 0x7f68d414d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41660/0 (socket says 172.21.15.186:41660) 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68d9312640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d40c2d00 0x7f68d40c5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:33216/0 (socket says 172.21.15.186:33216) 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68d9b13640 1 -- 172.21.15.186:0/1015342791 learned_addr learned my addr 172.21.15.186:0/1015342791 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68da314640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d4151990 0x7f68d40c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68d9b13640 1 -- 172.21.15.186:0/1015342791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d4151990 msgr2=0x7f68d40c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68d9b13640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d4151990 0x7f68d40c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68d9b13640 1 -- 172.21.15.186:0/1015342791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d40c2d00 msgr2=0x7f68d40c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68d9b13640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d40c2d00 0x7f68d40c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68d9b13640 1 -- 172.21.15.186:0/1015342791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d41f4960 con 0x7f68d40b1e30 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68d9b13640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d40b1e30 0x7f68d414d410 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f68d000c410 tx=0x7f68d000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68ca7fc640 1 -- 172.21.15.186:0/1015342791 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0017070 con 0x7f68d40b1e30 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68ca7fc640 1 -- 172.21.15.186:0/1015342791 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68d00035f0 con 0x7f68d40b1e30 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68dbd77640 1 -- 172.21.15.186:0/1015342791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d40ae980 con 0x7f68d40b1e30 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68ca7fc640 1 -- 172.21.15.186:0/1015342791 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d00039e0 con 0x7f68d40b1e30 2024-03-28T15:10:03.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.159+0000 7f68dbd77640 1 -- 172.21.15.186:0/1015342791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d40aef40 con 0x7f68d40b1e30 2024-03-28T15:10:03.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.161+0000 7f68dbd77640 1 -- 172.21.15.186:0/1015342791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d4072040 con 0x7f68d40b1e30 2024-03-28T15:10:03.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.161+0000 7f68ca7fc640 1 -- 172.21.15.186:0/1015342791 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f68d001a070 con 0x7f68d40b1e30 2024-03-28T15:10:03.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.161+0000 7f68ca7fc640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68b806c530 0x7f68b806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:03.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.161+0000 7f68ca7fc640 1 -- 172.21.15.186:0/1015342791 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f68d008ac50 con 0x7f68d40b1e30 2024-03-28T15:10:03.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.161+0000 7f68d9312640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68b806c530 0x7f68b806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:03.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.162+0000 7f68d9312640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68b806c530 0x7f68b806e9f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f68c4036000 tx=0x7f68c4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:03.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.166+0000 7f68ca7fc640 1 -- 172.21.15.186:0/1015342791 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f68d4072040 con 0x7f68d40b1e30 2024-03-28T15:10:03.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.379+0000 7f68dbd77640 1 -- 172.21.15.186:0/1015342791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68d4069500 con 0x7f68d40b1e30 2024-03-28T15:10:03.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.379+0000 7f68ca7fc640 1 -- 172.21.15.186:0/1015342791 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6012 (secure 0 0 0) 0x7f68d0090020 con 0x7f68d40b1e30 2024-03-28T15:10:03.380 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:03.380 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:09:56.237830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:10:03.380 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 30 2024-03-28T15:10:03.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 -- 172.21.15.186:0/1015342791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68b806c530 msgr2=0x7f68b806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:03.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68b806c530 0x7f68b806e9f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f68c4036000 tx=0x7f68c4008030 comp rx=0 tx=0).stop 2024-03-28T15:10:03.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 -- 172.21.15.186:0/1015342791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d40b1e30 msgr2=0x7f68d414d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:03.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d40b1e30 0x7f68d414d410 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f68d000c410 tx=0x7f68d000c8d0 comp rx=0 tx=0).stop 2024-03-28T15:10:03.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 -- 172.21.15.186:0/1015342791 shutdown_connections 2024-03-28T15:10:03.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f68b806c530 0x7f68b806e9f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:03.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d4151990 0x7f68d40c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:03.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d40c2d00 0x7f68d40c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:03.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 --2- 172.21.15.186:0/1015342791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d40b1e30 0x7f68d414d410 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:03.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 -- 172.21.15.186:0/1015342791 >> 172.21.15.186:0/1015342791 conn(0x7f68d40c8a20 msgr2=0x7f68d40c7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:03.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 -- 172.21.15.186:0/1015342791 shutdown_connections 2024-03-28T15:10:03.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:03.381+0000 7f68dbd77640 1 -- 172.21.15.186:0/1015342791 wait complete. 2024-03-28T15:10:03.393 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:10:05.394 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:05.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.503+0000 7fc2d5dc7640 1 Processor -- start 2024-03-28T15:10:05.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.503+0000 7fc2d5dc7640 1 -- start start 2024-03-28T15:10:05.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.503+0000 7fc2d5dc7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d0071340 0x7fc2d0158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:05.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.503+0000 7fc2d5dc7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d0072df0 con 0x7fc2d00c8090 2024-03-28T15:10:05.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2d5dc7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d00c8660 con 0x7fc2d0152810 2024-03-28T15:10:05.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2d5dc7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc2d0159500 con 0x7fc2d0071340 2024-03-28T15:10:05.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2cffff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fc2d00c8090 0x7fc2d00c6cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:46816/0 (socket says 172.21.15.186:46816) 2024-03-28T15:10:05.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2cffff640 1 -- 172.21.15.186:0/3073771888 learned_addr learned my addr 172.21.15.186:0/3073771888 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:05.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ceffd640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d0071340 0x7fc2d0158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:05.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2855628909 0 0) 0x7fc2d0072df0 con 0x7fc2d00c8090 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2b8003520 con 0x7fc2d00c8090 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4045865582 0 0) 0x7fc2d00c8660 con 0x7fc2d0152810 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d0072df0 con 0x7fc2d0152810 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3475158730 0 0) 0x7fc2b8003520 con 0x7fc2d00c8090 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d0071340 msgr2=0x7fc2d0158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d0071340 0x7fc2d0158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 >> v1:172.21.15.191:6789/0 conn(0x7fc2d0152810 legacy=0x7fc2d006c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d0071d50 con 0x7fc2d00c8090 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7fc2c80019f0 con 0x7fc2d00c8090 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ceffd640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d0071340 0x7fc2d0158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 >> v1:172.21.15.186:6789/0 conn(0x7fc2d00c8090 legacy=0x7fc2d00c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 0x7fc2b8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d0071340 0x7fc2b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b8004d40 0x7fc2b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:05.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0159500 con 0x7fc2b8003b70 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b800ba70 con 0x7fc2b8004d40 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b800bc70 con 0x7fc2d0071340 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.504+0000 7fc2cf7fe640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 0x7fc2b8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.505+0000 7fc2ceffd640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d0071340 0x7fc2b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.505+0000 7fc2cffff640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b8004d40 0x7fc2b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.505+0000 7fc2cf7fe640 1 -- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d0071340 msgr2=0x7fc2b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.505+0000 7fc2cf7fe640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d0071340 0x7fc2b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.505+0000 7fc2cf7fe640 1 -- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b8004d40 msgr2=0x7fc2b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.505+0000 7fc2cf7fe640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b8004d40 0x7fc2b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.505+0000 7fc2cf7fe640 1 -- 172.21.15.186:0/3073771888 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b8003520 con 0x7fc2b8003b70 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.505+0000 7fc2cf7fe640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 0x7fc2b8003f70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00018d0 tx=0x7fc2b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67059173a138f9f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.506+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c0037070 con 0x7fc2b8003b70 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.506+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c00052d0 con 0x7fc2b8003b70 2024-03-28T15:10:05.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.506+0000 7fc2ce7fc640 1 -- 172.21.15.186:0/3073771888 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00014f0 con 0x7fc2b8003b70 2024-03-28T15:10:05.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.506+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/3073771888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 msgr2=0x7fc2b8003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:05.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.506+0000 7fc2d5dc7640 1 --2- 172.21.15.186:0/3073771888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 0x7fc2b8003f70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00018d0 tx=0x7fc2b8005830 comp rx=0 tx=0).stop 2024-03-28T15:10:05.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.506+0000 7fc2cffff640 1 -- 172.21.15.186:0/3073771888 reap_dead start 2024-03-28T15:10:05.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.506+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/3073771888 shutdown_connections 2024-03-28T15:10:05.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.506+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/3073771888 >> 172.21.15.186:0/3073771888 conn(0x7fc2d00c8a20 msgr2=0x7fc2d00c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:05.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.506+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/3073771888 shutdown_connections 2024-03-28T15:10:05.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.506+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/3073771888 wait complete. 2024-03-28T15:10:05.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.507+0000 7fc2d5dc7640 1 Processor -- start 2024-03-28T15:10:05.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2d5dc7640 1 -- start start 2024-03-28T15:10:05.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2d5dc7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 0x7fc2d014e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2d5dc7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2d01ef040 0x7fc2d0158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2d5dc7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d014cbf0 0x7fc2d014a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2d5dc7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d00725f0 con 0x7fc2b8003b70 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2d5dc7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0072310 con 0x7fc2d01ef040 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2d5dc7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fc2d00728d0 con 0x7fc2d014cbf0 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2cf7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 0x7fc2d014e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2cf7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 0x7fc2d014e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41668/0 (socket says 172.21.15.186:41668) 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2cf7fe640 1 -- 172.21.15.186:0/2641882426 learned_addr learned my addr 172.21.15.186:0/2641882426 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2ceffd640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2d01ef040 0x7fc2d0158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.508+0000 7fc2cffff640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d014cbf0 0x7fc2d014a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.509+0000 7fc2cf7fe640 1 -- 172.21.15.186:0/2641882426 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d014cbf0 msgr2=0x7fc2d014a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.509+0000 7fc2cf7fe640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d014cbf0 0x7fc2d014a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.509+0000 7fc2cf7fe640 1 -- 172.21.15.186:0/2641882426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2d01ef040 msgr2=0x7fc2d0158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.509+0000 7fc2cf7fe640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2d01ef040 0x7fc2d0158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:05.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.509+0000 7fc2cf7fe640 1 -- 172.21.15.186:0/2641882426 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d01591e0 con 0x7fc2b8003b70 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.509+0000 7fc2cf7fe640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 0x7fc2d014e330 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00019f0 tx=0x7fc2c00012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.509+0000 7fc2b7fff640 1 -- 172.21.15.186:0/2641882426 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c0037070 con 0x7fc2b8003b70 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.509+0000 7fc2b7fff640 1 -- 172.21.15.186:0/2641882426 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c0038040 con 0x7fc2b8003b70 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.509+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/2641882426 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d0157910 con 0x7fc2b8003b70 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.509+0000 7fc2b7fff640 1 -- 172.21.15.186:0/2641882426 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00014f0 con 0x7fc2b8003b70 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.509+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/2641882426 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d0157e90 con 0x7fc2b8003b70 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.510+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/2641882426 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d00c6040 con 0x7fc2b8003b70 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.511+0000 7fc2b7fff640 1 -- 172.21.15.186:0/2641882426 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc2c00052d0 con 0x7fc2b8003b70 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.511+0000 7fc2b7fff640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc2bc06c560 0x7fc2bc06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.511+0000 7fc2b7fff640 1 -- 172.21.15.186:0/2641882426 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fc2c00acf30 con 0x7fc2b8003b70 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.511+0000 7fc2ceffd640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc2bc06c560 0x7fc2bc06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:05.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.511+0000 7fc2ceffd640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc2bc06c560 0x7fc2bc06ea20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc2d014d650 tx=0x7fc2c4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:05.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.515+0000 7fc2b7fff640 1 -- 172.21.15.186:0/2641882426 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fc2d00c6040 con 0x7fc2b8003b70 2024-03-28T15:10:05.729 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.729+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/2641882426 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2d0069500 con 0x7fc2b8003b70 2024-03-28T15:10:05.730 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.729+0000 7fc2b7fff640 1 -- 172.21.15.186:0/2641882426 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6016 (secure 0 0 0) 0x7fc2c007ace0 con 0x7fc2b8003b70 2024-03-28T15:10:05.730 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:05.730 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:04.601367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:10:05.730 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 31 2024-03-28T15:10:05.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.731+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/2641882426 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc2bc06c560 msgr2=0x7fc2bc06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:05.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.731+0000 7fc2d5dc7640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc2bc06c560 0x7fc2bc06ea20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc2d014d650 tx=0x7fc2c4009190 comp rx=0 tx=0).stop 2024-03-28T15:10:05.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.731+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/2641882426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 msgr2=0x7fc2d014e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:05.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.731+0000 7fc2d5dc7640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 0x7fc2d014e330 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00019f0 tx=0x7fc2c00012d0 comp rx=0 tx=0).stop 2024-03-28T15:10:05.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.731+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/2641882426 shutdown_connections 2024-03-28T15:10:05.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.731+0000 7fc2d5dc7640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fc2bc06c560 0x7fc2bc06ea20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:05.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.731+0000 7fc2d5dc7640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2d014cbf0 0x7fc2d014a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:05.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.731+0000 7fc2d5dc7640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2d01ef040 0x7fc2d0158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:05.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.731+0000 7fc2d5dc7640 1 --2- 172.21.15.186:0/2641882426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2b8003b70 0x7fc2d014e330 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:05.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.731+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/2641882426 >> 172.21.15.186:0/2641882426 conn(0x7fc2d00c8a20 msgr2=0x7fc2d00c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:05.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.732+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/2641882426 shutdown_connections 2024-03-28T15:10:05.733 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:05.732+0000 7fc2d5dc7640 1 -- 172.21.15.186:0/2641882426 wait complete. 2024-03-28T15:10:05.743 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:10:07.744 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:07.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.856+0000 7f528da5c640 1 Processor -- start 2024-03-28T15:10:07.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.856+0000 7f528da5c640 1 -- start start 2024-03-28T15:10:07.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.857+0000 7f528da5c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5288152810 0x7f5288158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:07.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.857+0000 7f528da5c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52880c8090 0x7f52880c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:07.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.857+0000 7f528da5c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5288072df0 con 0x7f5288071340 2024-03-28T15:10:07.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.857+0000 7f528da5c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f5288159500 con 0x7f52880c8090 2024-03-28T15:10:07.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.857+0000 7f528da5c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f52880c8660 con 0x7f5288152810 2024-03-28T15:10:07.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.857+0000 7f5286ffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5288152810 0x7f5288158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:07.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.857+0000 7f52877fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52880c8090 0x7f52880c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:07.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.857+0000 7f5286ffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5288152810 0x7f5288158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:56582/0 (socket says 172.21.15.186:56582) 2024-03-28T15:10:07.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.857+0000 7f5286ffd640 1 -- 172.21.15.186:0/240425461 learned_addr learned my addr 172.21.15.186:0/240425461 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:07.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.857+0000 7f52877fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52880c8090 0x7f52880c6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:33250/0 (socket says 172.21.15.186:33250) 2024-03-28T15:10:07.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.857+0000 7f52867fc640 1 --1- 172.21.15.186:0/240425461 >> v1:172.21.15.191:6789/0 conn(0x7f5288071340 0x7f528806c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:51842/0 (socket says 172.21.15.186:51842) 2024-03-28T15:10:07.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.858+0000 7f5286ffd640 1 -- 172.21.15.186:0/240425461 >> v2:172.21.15.191:3300/0 conn(0x7f52880c8090 msgr2=0x7f52880c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:07.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.858+0000 7f5286ffd640 1 --2- 172.21.15.186:0/240425461 >> v2:172.21.15.191:3300/0 conn(0x7f52880c8090 0x7f52880c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:07.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.858+0000 7f5286ffd640 1 -- 172.21.15.186:0/240425461 >> v1:172.21.15.191:6789/0 conn(0x7f5288071340 legacy=0x7f528806c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:07.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.858+0000 7f5285ffb640 1 -- 172.21.15.186:0/240425461 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1484543700 0 0) 0x7f5288072df0 con 0x7f5288071340 2024-03-28T15:10:07.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.858+0000 7f5286ffd640 1 -- 172.21.15.186:0/240425461 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5288071d50 con 0x7f5288152810 2024-03-28T15:10:07.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.858+0000 7f52877fe640 1 --2- 172.21.15.186:0/240425461 >> v2:172.21.15.191:3300/0 conn(0x7f52880c8090 0x7f52880c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:07.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f5286ffd640 1 --2- 172.21.15.186:0/240425461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5288152810 0x7f5288158fc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5278009e50 tx=0x7f527802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65012c09cbe14718 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:07.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f5285ffb640 1 -- 172.21.15.186:0/240425461 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527802cb00 con 0x7f5288152810 2024-03-28T15:10:07.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f5285ffb640 1 -- 172.21.15.186:0/240425461 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f527802cca0 con 0x7f5288152810 2024-03-28T15:10:07.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f5285ffb640 1 -- 172.21.15.186:0/240425461 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5278034610 con 0x7f5288152810 2024-03-28T15:10:07.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f528da5c640 1 -- 172.21.15.186:0/240425461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5288152810 msgr2=0x7f5288158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:07.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f528da5c640 1 --2- 172.21.15.186:0/240425461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5288152810 0x7f5288158fc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5278009e50 tx=0x7f527802bf60 comp rx=0 tx=0).stop 2024-03-28T15:10:07.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f528da5c640 1 -- 172.21.15.186:0/240425461 shutdown_connections 2024-03-28T15:10:07.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f528da5c640 1 --2- 172.21.15.186:0/240425461 >> v2:172.21.15.191:3300/0 conn(0x7f52880c8090 0x7f52880c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:07.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f528da5c640 1 --2- 172.21.15.186:0/240425461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5288152810 0x7f5288158fc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:07.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f528da5c640 1 -- 172.21.15.186:0/240425461 >> 172.21.15.186:0/240425461 conn(0x7f52880c8a20 msgr2=0x7f52880c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:07.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f528da5c640 1 -- 172.21.15.186:0/240425461 shutdown_connections 2024-03-28T15:10:07.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.859+0000 7f528da5c640 1 -- 172.21.15.186:0/240425461 wait complete. 2024-03-28T15:10:07.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.860+0000 7f528da5c640 1 Processor -- start 2024-03-28T15:10:07.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.861+0000 7f528da5c640 1 -- start start 2024-03-28T15:10:07.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.861+0000 7f528da5c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5288071340 0x7f528814fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:07.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.861+0000 7f528da5c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52880c8090 0x7f528814d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:07.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.861+0000 7f528da5c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5288152810 0x7f52881503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:07.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.861+0000 7f528da5c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52880725f0 con 0x7f5288152810 2024-03-28T15:10:07.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.861+0000 7f528da5c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5288072310 con 0x7f5288071340 2024-03-28T15:10:07.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.861+0000 7f528da5c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f52880728d0 con 0x7f52880c8090 2024-03-28T15:10:07.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.861+0000 7f52877fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5288152810 0x7f52881503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f52877fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5288152810 0x7f52881503a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41672/0 (socket says 172.21.15.186:41672) 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f52877fe640 1 -- 172.21.15.186:0/883869384 learned_addr learned my addr 172.21.15.186:0/883869384 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f52867fc640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52880c8090 0x7f528814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f5286ffd640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5288071340 0x7f528814fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f52877fe640 1 -- 172.21.15.186:0/883869384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52880c8090 msgr2=0x7f528814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f52877fe640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52880c8090 0x7f528814d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f52877fe640 1 -- 172.21.15.186:0/883869384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5288071340 msgr2=0x7f528814fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f52877fe640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5288071340 0x7f528814fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f52877fe640 1 -- 172.21.15.186:0/883869384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5288158fa0 con 0x7f5288152810 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f52877fe640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5288152810 0x7f52881503a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f528000e8b0 tx=0x7f528000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f526ffff640 1 -- 172.21.15.186:0/883869384 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5280017070 con 0x7f5288152810 2024-03-28T15:10:07.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f526ffff640 1 -- 172.21.15.186:0/883869384 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5280004020 con 0x7f5288152810 2024-03-28T15:10:07.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f526ffff640 1 -- 172.21.15.186:0/883869384 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52800043f0 con 0x7f5288152810 2024-03-28T15:10:07.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f528da5c640 1 -- 172.21.15.186:0/883869384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5288158d00 con 0x7f5288152810 2024-03-28T15:10:07.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.862+0000 7f528da5c640 1 -- 172.21.15.186:0/883869384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f528814e950 con 0x7f5288152810 2024-03-28T15:10:07.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.863+0000 7f528da5c640 1 -- 172.21.15.186:0/883869384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5288069500 con 0x7f5288152810 2024-03-28T15:10:07.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.864+0000 7f526ffff640 1 -- 172.21.15.186:0/883869384 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f528000c940 con 0x7f5288152810 2024-03-28T15:10:07.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.864+0000 7f526ffff640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f527406c530 0x7f527406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:07.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.864+0000 7f526ffff640 1 -- 172.21.15.186:0/883869384 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f528008b610 con 0x7f5288152810 2024-03-28T15:10:07.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.864+0000 7f5286ffd640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f527406c530 0x7f527406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:07.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.865+0000 7f5286ffd640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f527406c530 0x7f527406e9f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5278035000 tx=0x7f5278030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:07.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:07.869+0000 7f526ffff640 1 -- 172.21.15.186:0/883869384 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5280059450 con 0x7f5288152810 2024-03-28T15:10:08.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.080+0000 7f528da5c640 1 -- 172.21.15.186:0/883869384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52880c77f0 con 0x7f5288152810 2024-03-28T15:10:08.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.080+0000 7f526ffff640 1 -- 172.21.15.186:0/883869384 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6016 (secure 0 0 0) 0x7f528005d750 con 0x7f5288152810 2024-03-28T15:10:08.081 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:08.081 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:04.601367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:10:08.081 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 31 2024-03-28T15:10:08.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.082+0000 7f528da5c640 1 -- 172.21.15.186:0/883869384 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f527406c530 msgr2=0x7f527406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:08.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.082+0000 7f528da5c640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f527406c530 0x7f527406e9f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5278035000 tx=0x7f5278030000 comp rx=0 tx=0).stop 2024-03-28T15:10:08.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.082+0000 7f528da5c640 1 -- 172.21.15.186:0/883869384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5288152810 msgr2=0x7f52881503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:08.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.082+0000 7f528da5c640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5288152810 0x7f52881503a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f528000e8b0 tx=0x7f528000ed70 comp rx=0 tx=0).stop 2024-03-28T15:10:08.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.082+0000 7f528da5c640 1 -- 172.21.15.186:0/883869384 shutdown_connections 2024-03-28T15:10:08.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.082+0000 7f528da5c640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f527406c530 0x7f527406e9f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:08.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.082+0000 7f528da5c640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5288152810 0x7f52881503a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:08.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.082+0000 7f528da5c640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52880c8090 0x7f528814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:08.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.082+0000 7f528da5c640 1 --2- 172.21.15.186:0/883869384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5288071340 0x7f528814fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:08.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.082+0000 7f528da5c640 1 -- 172.21.15.186:0/883869384 >> 172.21.15.186:0/883869384 conn(0x7f52880c8a20 msgr2=0x7f52880cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:08.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.082+0000 7f528da5c640 1 -- 172.21.15.186:0/883869384 shutdown_connections 2024-03-28T15:10:08.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:08.083+0000 7f528da5c640 1 -- 172.21.15.186:0/883869384 wait complete. 2024-03-28T15:10:08.095 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:10:10.096 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:10.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.209+0000 7f3ce550c640 1 Processor -- start 2024-03-28T15:10:10.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.209+0000 7f3ce550c640 1 -- start start 2024-03-28T15:10:10.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.209+0000 7f3ce550c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3ce006be00 0x7f3ce00c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:10.210 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.209+0000 7f3ce550c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce015b120 con 0x7f3ce00c9b10 2024-03-28T15:10:10.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.210+0000 7f3ce550c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce006c870 con 0x7f3ce00c95e0 2024-03-28T15:10:10.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.210+0000 7f3ce550c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce006b540 con 0x7f3ce006be00 2024-03-28T15:10:10.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.210+0000 7f3cdf7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f3ce00c9b10 0x7f3ce01652b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:47746/0 (socket says 172.21.15.186:47746) 2024-03-28T15:10:10.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.210+0000 7f3cdeffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3ce006be00 0x7f3ce00c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:10.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.210+0000 7f3cdf7fe640 1 -- 172.21.15.186:0/856038303 learned_addr learned my addr 172.21.15.186:0/856038303 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:10.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.210+0000 7f3cdeffd640 1 -- 172.21.15.186:0/856038303 >> v1:172.21.15.191:6789/0 conn(0x7f3ce00c95e0 legacy=0x7f3ce015aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:10.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.210+0000 7f3cdeffd640 1 -- 172.21.15.186:0/856038303 >> v1:172.21.15.186:6789/0 conn(0x7f3ce00c9b10 legacy=0x7f3ce01652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:10.211 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.210+0000 7f3cdeffd640 1 -- 172.21.15.186:0/856038303 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce006cb50 con 0x7f3ce006be00 2024-03-28T15:10:10.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.211+0000 7f3cdeffd640 1 --2- 172.21.15.186:0/856038303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce006be00 0x7f3ce00c7a80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00097a0 tx=0x7f3cd002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cea8d5ac2238127 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:10.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.211+0000 7f3cddffb640 1 -- 172.21.15.186:0/856038303 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd002f810 con 0x7f3ce006be00 2024-03-28T15:10:10.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.211+0000 7f3cddffb640 1 -- 172.21.15.186:0/856038303 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd002f9b0 con 0x7f3ce006be00 2024-03-28T15:10:10.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.211+0000 7f3cddffb640 1 -- 172.21.15.186:0/856038303 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd002fd80 con 0x7f3ce006be00 2024-03-28T15:10:10.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.211+0000 7f3ce550c640 1 -- 172.21.15.186:0/856038303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce006be00 msgr2=0x7f3ce00c7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:10.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.211+0000 7f3ce550c640 1 --2- 172.21.15.186:0/856038303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce006be00 0x7f3ce00c7a80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00097a0 tx=0x7f3cd002b9f0 comp rx=0 tx=0).stop 2024-03-28T15:10:10.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.211+0000 7f3ce550c640 1 -- 172.21.15.186:0/856038303 shutdown_connections 2024-03-28T15:10:10.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.211+0000 7f3ce550c640 1 --2- 172.21.15.186:0/856038303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce006be00 0x7f3ce00c7a80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:10.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.211+0000 7f3ce550c640 1 -- 172.21.15.186:0/856038303 >> 172.21.15.186:0/856038303 conn(0x7f3ce006e970 msgr2=0x7f3ce006ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:10.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.212+0000 7f3ce550c640 1 -- 172.21.15.186:0/856038303 shutdown_connections 2024-03-28T15:10:10.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.212+0000 7f3ce550c640 1 -- 172.21.15.186:0/856038303 wait complete. 2024-03-28T15:10:10.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.213+0000 7f3ce550c640 1 Processor -- start 2024-03-28T15:10:10.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.213+0000 7f3ce550c640 1 -- start start 2024-03-28T15:10:10.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.213+0000 7f3ce550c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ce006be00 0x7f3ce00c2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.213+0000 7f3ce550c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce00c95e0 0x7f3ce01f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3ce550c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce00c9b10 0x7f3ce0151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3ce550c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce015e540 con 0x7f3ce00c95e0 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3ce550c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce015e260 con 0x7f3ce006be00 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3ce550c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce015e820 con 0x7f3ce00c9b10 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cde7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce00c95e0 0x7f3ce01f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cde7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce00c95e0 0x7f3ce01f3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34650/0 (socket says 172.21.15.186:34650) 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cde7fc640 1 -- 172.21.15.186:0/2134596945 learned_addr learned my addr 172.21.15.186:0/2134596945 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cdeffd640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ce006be00 0x7f3ce00c2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cdf7fe640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce00c9b10 0x7f3ce0151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cde7fc640 1 -- 172.21.15.186:0/2134596945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce00c9b10 msgr2=0x7f3ce0151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cde7fc640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce00c9b10 0x7f3ce0151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cde7fc640 1 -- 172.21.15.186:0/2134596945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ce006be00 msgr2=0x7f3ce00c2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cde7fc640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ce006be00 0x7f3ce00c2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:10.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cde7fc640 1 -- 172.21.15.186:0/2134596945 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce01e3d20 con 0x7f3ce00c95e0 2024-03-28T15:10:10.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cde7fc640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce00c95e0 0x7f3ce01f3c70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c400 tx=0x7f3cd400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:10.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cc77fe640 1 -- 172.21.15.186:0/2134596945 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40052d0 con 0x7f3ce00c95e0 2024-03-28T15:10:10.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cc77fe640 1 -- 172.21.15.186:0/2134596945 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd4005490 con 0x7f3ce00c95e0 2024-03-28T15:10:10.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3ce550c640 1 -- 172.21.15.186:0/2134596945 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce01e4000 con 0x7f3ce00c95e0 2024-03-28T15:10:10.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3cc77fe640 1 -- 172.21.15.186:0/2134596945 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4007260 con 0x7f3ce00c95e0 2024-03-28T15:10:10.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.214+0000 7f3ce550c640 1 -- 172.21.15.186:0/2134596945 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce01f0cd0 con 0x7f3ce00c95e0 2024-03-28T15:10:10.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.215+0000 7f3ce550c640 1 -- 172.21.15.186:0/2134596945 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce00708b0 con 0x7f3ce00c95e0 2024-03-28T15:10:10.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.216+0000 7f3cc77fe640 1 -- 172.21.15.186:0/2134596945 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3cd4007400 con 0x7f3ce00c95e0 2024-03-28T15:10:10.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.216+0000 7f3cc77fe640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ccc06c530 0x7f3ccc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:10.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.217+0000 7f3cdeffd640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ccc06c530 0x7f3ccc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:10.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.217+0000 7f3cc77fe640 1 -- 172.21.15.186:0/2134596945 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f3cd408add0 con 0x7f3ce00c95e0 2024-03-28T15:10:10.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.217+0000 7f3cdeffd640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ccc06c530 0x7f3ccc06e9f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0036000 tx=0x7f3cd0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:10.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.221+0000 7f3cc77fe640 1 -- 172.21.15.186:0/2134596945 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3ce00708b0 con 0x7f3ce00c95e0 2024-03-28T15:10:10.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.437+0000 7f3ce550c640 1 -- 172.21.15.186:0/2134596945 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ce0069500 con 0x7f3ce00c95e0 2024-03-28T15:10:10.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.438+0000 7f3cc77fe640 1 -- 172.21.15.186:0/2134596945 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6016 (secure 0 0 0) 0x7f3cd408c090 con 0x7f3ce00c95e0 2024-03-28T15:10:10.438 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:10.439 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:04.601367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:10:10.439 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 31 2024-03-28T15:10:10.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 -- 172.21.15.186:0/2134596945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ccc06c530 msgr2=0x7f3ccc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:10.440 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ccc06c530 0x7f3ccc06e9f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0036000 tx=0x7f3cd0008030 comp rx=0 tx=0).stop 2024-03-28T15:10:10.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 -- 172.21.15.186:0/2134596945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce00c95e0 msgr2=0x7f3ce01f3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:10.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce00c95e0 0x7f3ce01f3c70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c400 tx=0x7f3cd400c8c0 comp rx=0 tx=0).stop 2024-03-28T15:10:10.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 -- 172.21.15.186:0/2134596945 shutdown_connections 2024-03-28T15:10:10.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3ccc06c530 0x7f3ccc06e9f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:10.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ce00c9b10 0x7f3ce0151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:10.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce00c95e0 0x7f3ce01f3c70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:10.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 --2- 172.21.15.186:0/2134596945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ce006be00 0x7f3ce00c2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:10.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 -- 172.21.15.186:0/2134596945 >> 172.21.15.186:0/2134596945 conn(0x7f3ce006e970 msgr2=0x7f3ce00c7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:10.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 -- 172.21.15.186:0/2134596945 shutdown_connections 2024-03-28T15:10:10.441 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:10.440+0000 7f3ce550c640 1 -- 172.21.15.186:0/2134596945 wait complete. 2024-03-28T15:10:10.452 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 31, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T15:01:31.425299+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 31, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5278": { "addr": "172.21.15.191:6834/3646031339", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 3646031339, "type": "v2" }, { "addr": "172.21.15.191:6834", "nonce": 3646031339, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5278, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T15:10:04.601367+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5278 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/1278716016", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6833", "nonce": 1278716016, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1278716016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6835/2068761001", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2068761001, "type": "v2" }, { "addr": "172.21.15.186:6835", "nonce": 2068761001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2344737463", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6837", "nonce": 2344737463, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2344737463, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4284, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/1548903381", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 1548903381, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 1548903381, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T15:10:12.453 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:12.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.564+0000 7fdfa1ed4640 1 Processor -- start 2024-03-28T15:10:12.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.564+0000 7fdfa1ed4640 1 -- start start 2024-03-28T15:10:12.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdfa1ed4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdf9c154ae0 0x7fdf9c0c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:12.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdfa1ed4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf9c0c11f0 0x7fdf9c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:12.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdfa1ed4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf9c154ec0 con 0x7fdf9c15b1e0 2024-03-28T15:10:12.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdfa1ed4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdf9c06ce60 con 0x7fdf9c154ae0 2024-03-28T15:10:12.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdfa1ed4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9c0c82a0 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdf9b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdf9c154ae0 0x7fdf9c0c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdf9b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdf9c154ae0 0x7fdf9c0c29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34658/0 (socket says 172.21.15.186:34658) 2024-03-28T15:10:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdf9b7fe640 1 -- 172.21.15.186:0/59526076 learned_addr learned my addr 172.21.15.186:0/59526076 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdf9affd640 1 --2- 172.21.15.186:0/59526076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf9c0c11f0 0x7fdf9c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdf9b7fe640 1 -- 172.21.15.186:0/59526076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf9c0c11f0 msgr2=0x7fdf9c06c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdf9b7fe640 1 --2- 172.21.15.186:0/59526076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf9c0c11f0 0x7fdf9c06c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdf9b7fe640 1 -- 172.21.15.186:0/59526076 >> v1:172.21.15.191:6789/0 conn(0x7fdf9c15b1e0 legacy=0x7fdf9c0c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.565+0000 7fdf9b7fe640 1 -- 172.21.15.186:0/59526076 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c0c8540 con 0x7fdf9c154ae0 2024-03-28T15:10:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.566+0000 7fdf9b7fe640 1 --2- 172.21.15.186:0/59526076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf9c154ae0 0x7fdf9c0c29e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009870 tx=0x7fdf9002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5aac48c25f35ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.566+0000 7fdf9a7fc640 1 -- 172.21.15.186:0/59526076 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9002f840 con 0x7fdf9c154ae0 2024-03-28T15:10:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.566+0000 7fdf9a7fc640 1 -- 172.21.15.186:0/59526076 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9002f9e0 con 0x7fdf9c154ae0 2024-03-28T15:10:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.566+0000 7fdf9a7fc640 1 -- 172.21.15.186:0/59526076 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9002fdb0 con 0x7fdf9c154ae0 2024-03-28T15:10:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.567+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/59526076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf9c154ae0 msgr2=0x7fdf9c0c29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.567+0000 7fdfa1ed4640 1 --2- 172.21.15.186:0/59526076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf9c154ae0 0x7fdf9c0c29e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009870 tx=0x7fdf9002ba20 comp rx=0 tx=0).stop 2024-03-28T15:10:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.567+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/59526076 shutdown_connections 2024-03-28T15:10:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.567+0000 7fdfa1ed4640 1 --2- 172.21.15.186:0/59526076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf9c0c11f0 0x7fdf9c06c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.567+0000 7fdfa1ed4640 1 --2- 172.21.15.186:0/59526076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf9c154ae0 0x7fdf9c0c29e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.567+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/59526076 >> 172.21.15.186:0/59526076 conn(0x7fdf9c0c8a20 msgr2=0x7fdf9c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.567+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/59526076 shutdown_connections 2024-03-28T15:10:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.567+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/59526076 wait complete. 2024-03-28T15:10:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.568+0000 7fdfa1ed4640 1 Processor -- start 2024-03-28T15:10:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.568+0000 7fdfa1ed4640 1 -- start start 2024-03-28T15:10:12.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdfa1ed4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf9c0c11f0 0x7fdf9c06f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:12.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdfa1ed4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf9c154ae0 0x7fdf9c1585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:12.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdfa1ed4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf9c15b1e0 0x7fdf9c1e61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:12.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdfa1ed4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c0c1f00 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdfa1ed4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c0c1c20 con 0x7fdf9c15b1e0 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdfa1ed4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9c0c21e0 con 0x7fdf9c154ae0 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf9c0c11f0 0x7fdf9c06f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf9c0c11f0 0x7fdf9c06f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34660/0 (socket says 172.21.15.186:34660) 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9b7fe640 1 -- 172.21.15.186:0/2317952432 learned_addr learned my addr 172.21.15.186:0/2317952432 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9affd640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf9c154ae0 0x7fdf9c1585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9bfff640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf9c15b1e0 0x7fdf9c1e61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9b7fe640 1 -- 172.21.15.186:0/2317952432 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf9c154ae0 msgr2=0x7fdf9c1585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9b7fe640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf9c154ae0 0x7fdf9c1585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9b7fe640 1 -- 172.21.15.186:0/2317952432 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf9c15b1e0 msgr2=0x7fdf9c1e61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9b7fe640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf9c15b1e0 0x7fdf9c1e61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9b7fe640 1 -- 172.21.15.186:0/2317952432 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c152ad0 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9affd640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf9c154ae0 0x7fdf9c1585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:10:12.573 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9bfff640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf9c15b1e0 0x7fdf9c1e61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:10:12.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf9b7fe640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf9c0c11f0 0x7fdf9c06f7a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdf90000c00 tx=0x7fdf90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:12.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.569+0000 7fdf83fff640 1 -- 172.21.15.186:0/2317952432 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90037bc0 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.570+0000 7fdf83fff640 1 -- 172.21.15.186:0/2317952432 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf90046070 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.570+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/2317952432 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9c1f3d30 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.570+0000 7fdf83fff640 1 -- 172.21.15.186:0/2317952432 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90041400 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.570+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/2317952432 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9c1f42d0 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.571+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/2317952432 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf9c069500 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.571+0000 7fdf83fff640 1 -- 172.21.15.186:0/2317952432 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdf90040070 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.571+0000 7fdf83fff640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf8806c530 0x7fdf8806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:12.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.572+0000 7fdf83fff640 1 -- 172.21.15.186:0/2317952432 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fdf900adf50 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.572+0000 7fdf9affd640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf8806c530 0x7fdf8806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:12.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.572+0000 7fdf9affd640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf8806c530 0x7fdf8806e9f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c150190 tx=0x7fdf8c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:12.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.576+0000 7fdf83fff640 1 -- 172.21.15.186:0/2317952432 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fdf90033030 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.788+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/2317952432 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf9c152e10 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.789+0000 7fdf83fff640 1 -- 172.21.15.186:0/2317952432 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6013 (secure 0 0 0) 0x7fdf9007c350 con 0x7fdf9c0c11f0 2024-03-28T15:10:12.790 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:12.790 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:10.314259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:10:12.791 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 32 2024-03-28T15:10:12.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.790+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/2317952432 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf8806c530 msgr2=0x7fdf8806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:12.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.790+0000 7fdfa1ed4640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf8806c530 0x7fdf8806e9f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c150190 tx=0x7fdf8c008040 comp rx=0 tx=0).stop 2024-03-28T15:10:12.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.790+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/2317952432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf9c0c11f0 msgr2=0x7fdf9c06f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:12.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.790+0000 7fdfa1ed4640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf9c0c11f0 0x7fdf9c06f7a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdf90000c00 tx=0x7fdf90004730 comp rx=0 tx=0).stop 2024-03-28T15:10:12.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.790+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/2317952432 shutdown_connections 2024-03-28T15:10:12.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.790+0000 7fdfa1ed4640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdf8806c530 0x7fdf8806e9f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:12.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.790+0000 7fdfa1ed4640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf9c15b1e0 0x7fdf9c1e61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:12.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.790+0000 7fdfa1ed4640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf9c154ae0 0x7fdf9c1585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:12.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.790+0000 7fdfa1ed4640 1 --2- 172.21.15.186:0/2317952432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf9c0c11f0 0x7fdf9c06f7a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:12.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.791+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/2317952432 >> 172.21.15.186:0/2317952432 conn(0x7fdf9c0c8a20 msgr2=0x7fdf9c0c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:12.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.791+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/2317952432 shutdown_connections 2024-03-28T15:10:12.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:12.791+0000 7fdfa1ed4640 1 -- 172.21.15.186:0/2317952432 wait complete. 2024-03-28T15:10:12.804 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:10:14.804 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:14.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.917+0000 7f3b4421c640 1 Processor -- start 2024-03-28T15:10:14.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.917+0000 7f3b4421c640 1 -- start start 2024-03-28T15:10:14.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.917+0000 7f3b4421c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3b3c154bd0 0x7f3b3c15a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:14.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.917+0000 7f3b4421c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b3c0c6e00 con 0x7f3b3c0c57e0 2024-03-28T15:10:14.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.917+0000 7f3b4421c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b3c15b080 con 0x7f3b3c0c53c0 2024-03-28T15:10:14.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.917+0000 7f3b4421c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f3b3c0c5ca0 con 0x7f3b3c154bd0 2024-03-28T15:10:14.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b427b9640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f3b3c0c57e0 0x7f3b3c0cb040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:47748/0 (socket says 172.21.15.186:47748) 2024-03-28T15:10:14.919 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b427b9640 1 -- 172.21.15.186:0/3911215697 learned_addr learned my addr 172.21.15.186:0/3911215697 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:14.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148723924 0 0) 0x7f3b3c0c6e00 con 0x7f3b3c0c57e0 2024-03-28T15:10:14.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b41fb8640 1 --2- 172.21.15.186:0/3911215697 >> v2:172.21.15.191:3300/0 conn(0x7f3b3c154bd0 0x7f3b3c15a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:14.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b28003520 con 0x7f3b3c0c57e0 2024-03-28T15:10:14.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 556532731 0 0) 0x7f3b28003520 con 0x7f3b3c0c57e0 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 >> v2:172.21.15.191:3300/0 conn(0x7f3b3c154bd0 msgr2=0x7f3b3c15a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 --2- 172.21.15.186:0/3911215697 >> v2:172.21.15.191:3300/0 conn(0x7f3b3c154bd0 0x7f3b3c15a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 >> v1:172.21.15.191:6789/0 conn(0x7f3b3c0c53c0 legacy=0x7f3b3c0c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3c15b360 con 0x7f3b3c0c57e0 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 <== mon.0 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2517870046 0 0) 0x7f3b380019f0 con 0x7f3b3c0c57e0 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 >> v1:172.21.15.186:6789/0 conn(0x7f3b3c0c57e0 legacy=0x7f3b3c0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 --2- 172.21.15.186:0/3911215697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b280039b0 0x7f3b28003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 --2- 172.21.15.186:0/3911215697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b28004800 0x7f3b28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 --2- 172.21.15.186:0/3911215697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b280051c0 0x7f3b280141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c0c5ca0 con 0x7f3b28004800 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3b28014880 con 0x7f3b280051c0 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3b28014a80 con 0x7f3b280039b0 2024-03-28T15:10:14.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b417b7640 1 --2- 172.21.15.186:0/3911215697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b28004800 0x7f3b28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b427b9640 1 --2- 172.21.15.186:0/3911215697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b280051c0 0x7f3b280141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.918+0000 7f3b41fb8640 1 --2- 172.21.15.186:0/3911215697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b280039b0 0x7f3b28003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.919+0000 7f3b417b7640 1 -- 172.21.15.186:0/3911215697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b280039b0 msgr2=0x7f3b28003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.919+0000 7f3b417b7640 1 --2- 172.21.15.186:0/3911215697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b280039b0 0x7f3b28003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.919+0000 7f3b417b7640 1 -- 172.21.15.186:0/3911215697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b280051c0 msgr2=0x7f3b280141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.919+0000 7f3b417b7640 1 --2- 172.21.15.186:0/3911215697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b280051c0 0x7f3b280141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.919+0000 7f3b427b9640 1 -- 172.21.15.186:0/3911215697 reap_dead start 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.919+0000 7f3b417b7640 1 -- 172.21.15.186:0/3911215697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b28003520 con 0x7f3b28004800 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.919+0000 7f3b417b7640 1 --2- 172.21.15.186:0/3911215697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b28004800 0x7f3b28004c80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c001750 tx=0x7f3b28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f82569d6e32579 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.920+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c001370 con 0x7f3b28004800 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.920+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b2c0052d0 con 0x7f3b28004800 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.920+0000 7f3b40fb6640 1 -- 172.21.15.186:0/3911215697 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c003830 con 0x7f3b28004800 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.920+0000 7f3b4421c640 1 -- 172.21.15.186:0/3911215697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b28004800 msgr2=0x7f3b28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:14.923 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.920+0000 7f3b4421c640 1 --2- 172.21.15.186:0/3911215697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b28004800 0x7f3b28004c80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c001750 tx=0x7f3b28005830 comp rx=0 tx=0).stop 2024-03-28T15:10:14.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.920+0000 7f3b4421c640 1 -- 172.21.15.186:0/3911215697 shutdown_connections 2024-03-28T15:10:14.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.920+0000 7f3b4421c640 1 --2- 172.21.15.186:0/3911215697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b28004800 0x7f3b28004c80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:14.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.920+0000 7f3b4421c640 1 -- 172.21.15.186:0/3911215697 >> 172.21.15.186:0/3911215697 conn(0x7f3b3c0731f0 msgr2=0x7f3b3c073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:14.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.920+0000 7f3b4421c640 1 -- 172.21.15.186:0/3911215697 shutdown_connections 2024-03-28T15:10:14.924 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.920+0000 7f3b4421c640 1 -- 172.21.15.186:0/3911215697 wait complete. 2024-03-28T15:10:14.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.921+0000 7f3b4421c640 1 Processor -- start 2024-03-28T15:10:14.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.921+0000 7f3b4421c640 1 -- start start 2024-03-28T15:10:14.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.922+0000 7f3b4421c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b28004800 0x7f3b3c15e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:14.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.922+0000 7f3b4421c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b3c0c8420 0x7f3b3c14ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:14.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.922+0000 7f3b4421c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b3c1566f0 0x7f3b3c156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:14.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.922+0000 7f3b4421c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c0c61d0 con 0x7f3b3c1566f0 2024-03-28T15:10:14.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.922+0000 7f3b4421c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c0c5ef0 con 0x7f3b28004800 2024-03-28T15:10:14.929 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.922+0000 7f3b4421c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3b3c0cc390 con 0x7f3b3c0c8420 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.922+0000 7f3b427b9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b3c1566f0 0x7f3b3c156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.922+0000 7f3b427b9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b3c1566f0 0x7f3b3c156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34676/0 (socket says 172.21.15.186:34676) 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.922+0000 7f3b427b9640 1 -- 172.21.15.186:0/3913345180 learned_addr learned my addr 172.21.15.186:0/3913345180 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.922+0000 7f3b41fb8640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b28004800 0x7f3b3c15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.922+0000 7f3b417b7640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b3c0c8420 0x7f3b3c14ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.923+0000 7f3b427b9640 1 -- 172.21.15.186:0/3913345180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b3c0c8420 msgr2=0x7f3b3c14ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.923+0000 7f3b427b9640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b3c0c8420 0x7f3b3c14ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.923+0000 7f3b427b9640 1 -- 172.21.15.186:0/3913345180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b28004800 msgr2=0x7f3b3c15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.923+0000 7f3b427b9640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b28004800 0x7f3b3c15e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.923+0000 7f3b427b9640 1 -- 172.21.15.186:0/3913345180 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3c15d740 con 0x7f3b3c1566f0 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.923+0000 7f3b427b9640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b3c1566f0 0x7f3b3c156b70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c1583e0 tx=0x7f3b38007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.923+0000 7f3b327fc640 1 -- 172.21.15.186:0/3913345180 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3800aa80 con 0x7f3b3c1566f0 2024-03-28T15:10:14.930 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.923+0000 7f3b327fc640 1 -- 172.21.15.186:0/3913345180 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b38018070 con 0x7f3b3c1566f0 2024-03-28T15:10:14.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.923+0000 7f3b4421c640 1 -- 172.21.15.186:0/3913345180 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b3c15da00 con 0x7f3b3c1566f0 2024-03-28T15:10:14.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.923+0000 7f3b327fc640 1 -- 172.21.15.186:0/3913345180 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3800ae40 con 0x7f3b3c1566f0 2024-03-28T15:10:14.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.923+0000 7f3b4421c640 1 -- 172.21.15.186:0/3913345180 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b3c157540 con 0x7f3b3c1566f0 2024-03-28T15:10:14.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.924+0000 7f3b4421c640 1 -- 172.21.15.186:0/3913345180 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b3c069500 con 0x7f3b3c1566f0 2024-03-28T15:10:14.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.924+0000 7f3b327fc640 1 -- 172.21.15.186:0/3913345180 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3b3801d460 con 0x7f3b3c1566f0 2024-03-28T15:10:14.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.925+0000 7f3b327fc640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3b2006c530 0x7f3b2006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:14.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.925+0000 7f3b327fc640 1 -- 172.21.15.186:0/3913345180 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f3b3808bbf0 con 0x7f3b3c1566f0 2024-03-28T15:10:14.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.925+0000 7f3b41fb8640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3b2006c530 0x7f3b2006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:14.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.925+0000 7f3b41fb8640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3b2006c530 0x7f3b2006e9f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009bc0 tx=0x7f3b34009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:14.931 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:14.930+0000 7f3b327fc640 1 -- 172.21.15.186:0/3913345180 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3b3805a140 con 0x7f3b3c1566f0 2024-03-28T15:10:15.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.140+0000 7f3b4421c640 1 -- 172.21.15.186:0/3913345180 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b3c0cbb60 con 0x7f3b3c1566f0 2024-03-28T15:10:15.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.141+0000 7f3b327fc640 1 -- 172.21.15.186:0/3913345180 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6013 (secure 0 0 0) 0x7f3b38059a40 con 0x7f3b3c1566f0 2024-03-28T15:10:15.170 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:15.170 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:10.314259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:10:15.172 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 32 2024-03-28T15:10:15.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 -- 172.21.15.186:0/3913345180 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3b2006c530 msgr2=0x7f3b2006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:15.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3b2006c530 0x7f3b2006e9f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009bc0 tx=0x7f3b34009190 comp rx=0 tx=0).stop 2024-03-28T15:10:15.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 -- 172.21.15.186:0/3913345180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b3c1566f0 msgr2=0x7f3b3c156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:15.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b3c1566f0 0x7f3b3c156b70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c1583e0 tx=0x7f3b38007b10 comp rx=0 tx=0).stop 2024-03-28T15:10:15.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 -- 172.21.15.186:0/3913345180 shutdown_connections 2024-03-28T15:10:15.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3b2006c530 0x7f3b2006e9f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:15.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b3c1566f0 0x7f3b3c156b70 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:15.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b3c0c8420 0x7f3b3c14ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:15.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 --2- 172.21.15.186:0/3913345180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b28004800 0x7f3b3c15e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:15.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 -- 172.21.15.186:0/3913345180 >> 172.21.15.186:0/3913345180 conn(0x7f3b3c0731f0 msgr2=0x7f3b3c153e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:15.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 -- 172.21.15.186:0/3913345180 shutdown_connections 2024-03-28T15:10:15.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:15.143+0000 7f3b4421c640 1 -- 172.21.15.186:0/3913345180 wait complete. 2024-03-28T15:10:15.173 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:10:17.174 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:17.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.287+0000 7f32615b6640 1 Processor -- start 2024-03-28T15:10:17.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.287+0000 7f32615b6640 1 -- start start 2024-03-28T15:10:17.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.288+0000 7f32615b6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f325c0b1e30 0x7f325c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:17.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.288+0000 7f32615b6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f325c06caa0 con 0x7f325c151990 2024-03-28T15:10:17.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.288+0000 7f32615b6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f325c06cd80 con 0x7f325c0c2d00 2024-03-28T15:10:17.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.288+0000 7f32615b6640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f325c0c1da0 con 0x7f325c0b1e30 2024-03-28T15:10:17.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.288+0000 7f325a7fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f325c0b1e30 0x7f325c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:17.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.288+0000 7f325affd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f325c151990 0x7f325c0c2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:47762/0 (socket says 172.21.15.186:47762) 2024-03-28T15:10:17.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.288+0000 7f325affd640 1 -- 172.21.15.186:0/1020857836 learned_addr learned my addr 172.21.15.186:0/1020857836 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:17.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.288+0000 7f325a7fc640 1 -- 172.21.15.186:0/1020857836 >> v1:172.21.15.191:6789/0 conn(0x7f325c0c2d00 legacy=0x7f325c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:17.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.288+0000 7f325a7fc640 1 -- 172.21.15.186:0/1020857836 >> v1:172.21.15.186:6789/0 conn(0x7f325c151990 legacy=0x7f325c0c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:17.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.288+0000 7f3259ffb640 1 -- 172.21.15.186:0/1020857836 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2493822936 0 0) 0x7f325c06caa0 con 0x7f325c151990 2024-03-28T15:10:17.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.288+0000 7f325a7fc640 1 -- 172.21.15.186:0/1020857836 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325c0c1340 con 0x7f325c0b1e30 2024-03-28T15:10:17.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.289+0000 7f325a7fc640 1 --2- 172.21.15.186:0/1020857836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c0b1e30 0x7f325c0712b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f32500097a0 tx=0x7f325002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a60846e23ccd89 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:17.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.289+0000 7f3259ffb640 1 -- 172.21.15.186:0/1020857836 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250034070 con 0x7f325c0b1e30 2024-03-28T15:10:17.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.289+0000 7f3259ffb640 1 -- 172.21.15.186:0/1020857836 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3250030610 con 0x7f325c0b1e30 2024-03-28T15:10:17.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.289+0000 7f3259ffb640 1 -- 172.21.15.186:0/1020857836 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32500309e0 con 0x7f325c0b1e30 2024-03-28T15:10:17.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.290+0000 7f32615b6640 1 -- 172.21.15.186:0/1020857836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c0b1e30 msgr2=0x7f325c0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:17.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.290+0000 7f32615b6640 1 --2- 172.21.15.186:0/1020857836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c0b1e30 0x7f325c0712b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f32500097a0 tx=0x7f325002b960 comp rx=0 tx=0).stop 2024-03-28T15:10:17.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.290+0000 7f32615b6640 1 -- 172.21.15.186:0/1020857836 shutdown_connections 2024-03-28T15:10:17.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.290+0000 7f32615b6640 1 --2- 172.21.15.186:0/1020857836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c0b1e30 0x7f325c0712b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:17.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.290+0000 7f32615b6640 1 -- 172.21.15.186:0/1020857836 >> 172.21.15.186:0/1020857836 conn(0x7f325c0c8a20 msgr2=0x7f325c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:17.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.290+0000 7f32615b6640 1 -- 172.21.15.186:0/1020857836 shutdown_connections 2024-03-28T15:10:17.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.290+0000 7f32615b6640 1 -- 172.21.15.186:0/1020857836 wait complete. 2024-03-28T15:10:17.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.291+0000 7f32615b6640 1 Processor -- start 2024-03-28T15:10:17.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.291+0000 7f32615b6640 1 -- start start 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f32615b6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c0b1e30 0x7f325c158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f32615b6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f325c0c2d00 0x7f325c1ee270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f32615b6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f325c151990 0x7f325c154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f32615b6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f325c0c2940 con 0x7f325c0b1e30 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f32615b6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f325c0c1900 con 0x7f325c151990 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f32615b6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f325c0c2ac0 con 0x7f325c0c2d00 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f325affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c0b1e30 0x7f325c158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f325affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c0b1e30 0x7f325c158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34698/0 (socket says 172.21.15.186:34698) 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f325affd640 1 -- 172.21.15.186:0/3858742800 learned_addr learned my addr 172.21.15.186:0/3858742800 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f325affd640 1 -- 172.21.15.186:0/3858742800 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f325c0c2d00 msgr2=0x7f325c1ee270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f325b7fe640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f325c151990 0x7f325c154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f325a7fc640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f325c0c2d00 0x7f325c1ee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f325affd640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f325c0c2d00 0x7f325c1ee270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f325affd640 1 -- 172.21.15.186:0/3858742800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f325c151990 msgr2=0x7f325c154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:17.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f325affd640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f325c151990 0x7f325c154090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:17.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f325affd640 1 -- 172.21.15.186:0/3858742800 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325c1ee7b0 con 0x7f325c0b1e30 2024-03-28T15:10:17.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.292+0000 7f325a7fc640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f325c0c2d00 0x7f325c1ee270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:10:17.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.293+0000 7f325affd640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c0b1e30 0x7f325c158440 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f324c00c8c0 tx=0x7f324c00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:17.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.293+0000 7f32437fe640 1 -- 172.21.15.186:0/3858742800 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c018070 con 0x7f325c0b1e30 2024-03-28T15:10:17.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.293+0000 7f32437fe640 1 -- 172.21.15.186:0/3858742800 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f324c00a640 con 0x7f325c0b1e30 2024-03-28T15:10:17.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.293+0000 7f32615b6640 1 -- 172.21.15.186:0/3858742800 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f325c1f18a0 con 0x7f325c0b1e30 2024-03-28T15:10:17.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.293+0000 7f32437fe640 1 -- 172.21.15.186:0/3858742800 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c00aa30 con 0x7f325c0b1e30 2024-03-28T15:10:17.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.293+0000 7f32615b6640 1 -- 172.21.15.186:0/3858742800 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f325c1eda70 con 0x7f325c0b1e30 2024-03-28T15:10:17.296 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.294+0000 7f32615b6640 1 -- 172.21.15.186:0/3858742800 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f325c150be0 con 0x7f325c0b1e30 2024-03-28T15:10:17.296 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.294+0000 7f32437fe640 1 -- 172.21.15.186:0/3858742800 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f324c01b070 con 0x7f325c0b1e30 2024-03-28T15:10:17.296 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.295+0000 7f32437fe640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f324806c530 0x7f324806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:17.296 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.295+0000 7f32437fe640 1 -- 172.21.15.186:0/3858742800 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f324c08af90 con 0x7f325c0b1e30 2024-03-28T15:10:17.296 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.295+0000 7f325a7fc640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f324806c530 0x7f324806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:17.296 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.295+0000 7f325a7fc640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f324806c530 0x7f324806e9f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3250036000 tx=0x7f3250008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:17.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.299+0000 7f32437fe640 1 -- 172.21.15.186:0/3858742800 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f325c150be0 con 0x7f325c0b1e30 2024-03-28T15:10:17.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.514+0000 7f32615b6640 1 -- 172.21.15.186:0/3858742800 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f325c069500 con 0x7f325c0b1e30 2024-03-28T15:10:17.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.515+0000 7f32437fe640 1 -- 172.21.15.186:0/3858742800 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6019 (secure 0 0 0) 0x7f324c059390 con 0x7f325c0b1e30 2024-03-28T15:10:17.515 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:17.516 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:16.464120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:clientreplay","state_seq":23,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:10:17.516 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 33 2024-03-28T15:10:17.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 -- 172.21.15.186:0/3858742800 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f324806c530 msgr2=0x7f324806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:17.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f324806c530 0x7f324806e9f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3250036000 tx=0x7f3250008030 comp rx=0 tx=0).stop 2024-03-28T15:10:17.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 -- 172.21.15.186:0/3858742800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c0b1e30 msgr2=0x7f325c158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:17.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c0b1e30 0x7f325c158440 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f324c00c8c0 tx=0x7f324c00cd80 comp rx=0 tx=0).stop 2024-03-28T15:10:17.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 -- 172.21.15.186:0/3858742800 shutdown_connections 2024-03-28T15:10:17.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f324806c530 0x7f324806e9f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:17.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f325c151990 0x7f325c154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:17.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f325c0c2d00 0x7f325c1ee270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:17.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 --2- 172.21.15.186:0/3858742800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f325c0b1e30 0x7f325c158440 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:17.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 -- 172.21.15.186:0/3858742800 >> 172.21.15.186:0/3858742800 conn(0x7f325c0c8a20 msgr2=0x7f325c0c2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:17.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 -- 172.21.15.186:0/3858742800 shutdown_connections 2024-03-28T15:10:17.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:17.517+0000 7f32615b6640 1 -- 172.21.15.186:0/3858742800 wait complete. 2024-03-28T15:10:17.528 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:10:19.530 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:19.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.642+0000 7f5a9a581640 1 Processor -- start 2024-03-28T15:10:19.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.642+0000 7f5a9a581640 1 -- start start 2024-03-28T15:10:19.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.643+0000 7f5a9a581640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a940c95e0 0x7f5a9415aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:19.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.643+0000 7f5a9a581640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a9406c870 con 0x7f5a9406be00 2024-03-28T15:10:19.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.643+0000 7f5a9a581640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a9406cb50 con 0x7f5a940c9b10 2024-03-28T15:10:19.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.643+0000 7f5a9a581640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5a9406b540 con 0x7f5a940c95e0 2024-03-28T15:10:19.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.643+0000 7f5a93fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f5a9406be00 0x7f5a940c7a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:37588/0 (socket says 172.21.15.186:37588) 2024-03-28T15:10:19.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.643+0000 7f5a93fff640 1 -- 172.21.15.186:0/4201626451 learned_addr learned my addr 172.21.15.186:0/4201626451 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:19.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.643+0000 7f5a937fe640 1 --2- 172.21.15.186:0/4201626451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a940c95e0 0x7f5a9415aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:19.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.643+0000 7f5a92ffd640 1 -- 172.21.15.186:0/4201626451 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1093503250 0 0) 0x7f5a9406c870 con 0x7f5a9406be00 2024-03-28T15:10:19.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.643+0000 7f5a92ffd640 1 -- 172.21.15.186:0/4201626451 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a7c003520 con 0x7f5a9406be00 2024-03-28T15:10:19.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.644+0000 7f5a92ffd640 1 -- 172.21.15.186:0/4201626451 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3277633172 0 0) 0x7f5a9406cb50 con 0x7f5a940c9b10 2024-03-28T15:10:19.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.644+0000 7f5a937fe640 1 -- 172.21.15.186:0/4201626451 >> v1:172.21.15.191:6789/0 conn(0x7f5a940c9b10 legacy=0x7f5a941652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:19.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.644+0000 7f5a937fe640 1 -- 172.21.15.186:0/4201626451 >> v1:172.21.15.186:6789/0 conn(0x7f5a9406be00 legacy=0x7f5a940c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:19.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.644+0000 7f5a937fe640 1 -- 172.21.15.186:0/4201626451 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a9415df80 con 0x7f5a940c95e0 2024-03-28T15:10:19.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.644+0000 7f5a937fe640 1 --2- 172.21.15.186:0/4201626451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a940c95e0 0x7f5a9415aa10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5a880097d0 tx=0x7f5a8802b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0a213bdfef7b7e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:19.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.645+0000 7f5a92ffd640 1 -- 172.21.15.186:0/4201626451 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a88037070 con 0x7f5a940c95e0 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.645+0000 7f5a92ffd640 1 -- 172.21.15.186:0/4201626451 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a88030510 con 0x7f5a940c95e0 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.645+0000 7f5a92ffd640 1 -- 172.21.15.186:0/4201626451 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a84001670 con 0x7f5a940c95e0 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.645+0000 7f5a9a581640 1 -- 172.21.15.186:0/4201626451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a940c95e0 msgr2=0x7f5a9415aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.645+0000 7f5a9a581640 1 --2- 172.21.15.186:0/4201626451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a940c95e0 0x7f5a9415aa10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5a880097d0 tx=0x7f5a8802b7b0 comp rx=0 tx=0).stop 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.645+0000 7f5a9a581640 1 -- 172.21.15.186:0/4201626451 shutdown_connections 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.645+0000 7f5a9a581640 1 --2- 172.21.15.186:0/4201626451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a940c95e0 0x7f5a9415aa10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.645+0000 7f5a9a581640 1 -- 172.21.15.186:0/4201626451 >> 172.21.15.186:0/4201626451 conn(0x7f5a9406e970 msgr2=0x7f5a9406ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.645+0000 7f5a9a581640 1 -- 172.21.15.186:0/4201626451 shutdown_connections 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.645+0000 7f5a9a581640 1 -- 172.21.15.186:0/4201626451 wait complete. 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.646+0000 7f5a9a581640 1 Processor -- start 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.647+0000 7f5a9a581640 1 -- start start 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.647+0000 7f5a9a581640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a9406be00 0x7f5a941f1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.647+0000 7f5a9a581640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a940c95e0 0x7f5a940cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:19.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.647+0000 7f5a9a581640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a940c9b10 0x7f5a941534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.647+0000 7f5a9a581640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9415e820 con 0x7f5a940c95e0 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.647+0000 7f5a9a581640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9415e540 con 0x7f5a9406be00 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.647+0000 7f5a9a581640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5a9406c530 con 0x7f5a940c9b10 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.647+0000 7f5a937fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a940c95e0 0x7f5a940cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.647+0000 7f5a937fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a940c95e0 0x7f5a940cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54888/0 (socket says 172.21.15.186:54888) 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.647+0000 7f5a937fe640 1 -- 172.21.15.186:0/442781951 learned_addr learned my addr 172.21.15.186:0/442781951 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a93fff640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a9406be00 0x7f5a941f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a98b1e640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a940c9b10 0x7f5a941534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a937fe640 1 -- 172.21.15.186:0/442781951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a940c9b10 msgr2=0x7f5a941534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a937fe640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a940c9b10 0x7f5a941534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a937fe640 1 -- 172.21.15.186:0/442781951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a9406be00 msgr2=0x7f5a941f1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:19.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a937fe640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a9406be00 0x7f5a941f1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a937fe640 1 -- 172.21.15.186:0/442781951 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a9415c9d0 con 0x7f5a940c95e0 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a937fe640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a940c95e0 0x7f5a940cb0f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5a8802b7f0 tx=0x7f5a880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a90ff9640 1 -- 172.21.15.186:0/442781951 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a88042070 con 0x7f5a940c95e0 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a90ff9640 1 -- 172.21.15.186:0/442781951 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a8803e930 con 0x7f5a940c95e0 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a9a581640 1 -- 172.21.15.186:0/442781951 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a940cbbe0 con 0x7f5a940c95e0 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a90ff9640 1 -- 172.21.15.186:0/442781951 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a88037040 con 0x7f5a940c95e0 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.648+0000 7f5a9a581640 1 -- 172.21.15.186:0/442781951 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a940cc1a0 con 0x7f5a940c95e0 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.649+0000 7f5a9a581640 1 -- 172.21.15.186:0/442781951 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a94069500 con 0x7f5a940c95e0 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.650+0000 7f5a90ff9640 1 -- 172.21.15.186:0/442781951 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5a88045070 con 0x7f5a940c95e0 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.650+0000 7f5a90ff9640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5a8006c560 0x7f5a8006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.650+0000 7f5a90ff9640 1 -- 172.21.15.186:0/442781951 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f5a880ad0b0 con 0x7f5a940c95e0 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.650+0000 7f5a93fff640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5a8006c560 0x7f5a8006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:19.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.651+0000 7f5a93fff640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5a8006c560 0x7f5a8006ea20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5a84000c00 tx=0x7f5a840048c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:19.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.655+0000 7f5a90ff9640 1 -- 172.21.15.186:0/442781951 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5a8804c070 con 0x7f5a940c95e0 2024-03-28T15:10:19.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.866+0000 7f5a9a581640 1 -- 172.21.15.186:0/442781951 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a9406c310 con 0x7f5a940c95e0 2024-03-28T15:10:19.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.867+0000 7f5a90ff9640 1 -- 172.21.15.186:0/442781951 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6019 (secure 0 0 0) 0x7f5a9415c9d0 con 0x7f5a940c95e0 2024-03-28T15:10:19.867 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:19.867 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:16.464120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:clientreplay","state_seq":23,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:10:19.868 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 33 2024-03-28T15:10:19.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 -- 172.21.15.186:0/442781951 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5a8006c560 msgr2=0x7f5a8006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:19.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5a8006c560 0x7f5a8006ea20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5a84000c00 tx=0x7f5a840048c0 comp rx=0 tx=0).stop 2024-03-28T15:10:19.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 -- 172.21.15.186:0/442781951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a940c95e0 msgr2=0x7f5a940cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:19.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a940c95e0 0x7f5a940cb0f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5a8802b7f0 tx=0x7f5a880046c0 comp rx=0 tx=0).stop 2024-03-28T15:10:19.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 -- 172.21.15.186:0/442781951 shutdown_connections 2024-03-28T15:10:19.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5a8006c560 0x7f5a8006ea20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:19.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a940c9b10 0x7f5a941534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:19.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a940c95e0 0x7f5a940cb0f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:19.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 --2- 172.21.15.186:0/442781951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a9406be00 0x7f5a941f1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:19.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 -- 172.21.15.186:0/442781951 >> 172.21.15.186:0/442781951 conn(0x7f5a9406e970 msgr2=0x7f5a940723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:19.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 -- 172.21.15.186:0/442781951 shutdown_connections 2024-03-28T15:10:19.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:19.869+0000 7f5a9a581640 1 -- 172.21.15.186:0/442781951 wait complete. 2024-03-28T15:10:19.880 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-28T15:10:21.881 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:21.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.993+0000 7fd0cb8ec640 1 Processor -- start 2024-03-28T15:10:21.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.993+0000 7fd0cb8ec640 1 -- start start 2024-03-28T15:10:21.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0cb8ec640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd0c4151990 0x7fd0c40c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:21.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0cb8ec640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd0c40b1e30 0x7fd0c40c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:21.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0cb8ec640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0c406caa0 con 0x7fd0c40c2d00 2024-03-28T15:10:21.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0cb8ec640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd0c40c1da0 con 0x7fd0c40b1e30 2024-03-28T15:10:21.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0cb8ec640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fd0c406cd80 con 0x7fd0c4151990 2024-03-28T15:10:21.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0c8e87640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd0c40b1e30 0x7fd0c40c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:21.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0c8e87640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd0c40b1e30 0x7fd0c40c2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54890/0 (socket says 172.21.15.186:54890) 2024-03-28T15:10:21.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0c8e87640 1 -- 172.21.15.186:0/4186574326 learned_addr learned my addr 172.21.15.186:0/4186574326 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:21.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0c9688640 1 --2- 172.21.15.186:0/4186574326 >> v2:172.21.15.191:3300/0 conn(0x7fd0c4151990 0x7fd0c40c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:21.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0c8e87640 1 -- 172.21.15.186:0/4186574326 >> v2:172.21.15.191:3300/0 conn(0x7fd0c4151990 msgr2=0x7fd0c40c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:21.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0c8e87640 1 --2- 172.21.15.186:0/4186574326 >> v2:172.21.15.191:3300/0 conn(0x7fd0c4151990 0x7fd0c40c2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:21.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0c8e87640 1 -- 172.21.15.186:0/4186574326 >> v1:172.21.15.191:6789/0 conn(0x7fd0c40c2d00 legacy=0x7fd0c40712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:21.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.994+0000 7fd0c8e87640 1 -- 172.21.15.186:0/4186574326 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0c40c1340 con 0x7fd0c40b1e30 2024-03-28T15:10:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.995+0000 7fd0c8e87640 1 --2- 172.21.15.186:0/4186574326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0c40b1e30 0x7fd0c40c2770 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd0c406fd90 tx=0x7fd0b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e830fce9af98584d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.995+0000 7fd0bbfff640 1 -- 172.21.15.186:0/4186574326 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4034070 con 0x7fd0c40b1e30 2024-03-28T15:10:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.995+0000 7fd0bbfff640 1 -- 172.21.15.186:0/4186574326 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0b4030860 con 0x7fd0c40b1e30 2024-03-28T15:10:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.995+0000 7fd0bbfff640 1 -- 172.21.15.186:0/4186574326 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4030c30 con 0x7fd0c40b1e30 2024-03-28T15:10:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.996+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/4186574326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0c40b1e30 msgr2=0x7fd0c40c2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.996+0000 7fd0cb8ec640 1 --2- 172.21.15.186:0/4186574326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0c40b1e30 0x7fd0c40c2770 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd0c406fd90 tx=0x7fd0b402b9a0 comp rx=0 tx=0).stop 2024-03-28T15:10:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.996+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/4186574326 shutdown_connections 2024-03-28T15:10:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.996+0000 7fd0cb8ec640 1 --2- 172.21.15.186:0/4186574326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0c40b1e30 0x7fd0c40c2770 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.996+0000 7fd0cb8ec640 1 --2- 172.21.15.186:0/4186574326 >> v2:172.21.15.191:3300/0 conn(0x7fd0c4151990 0x7fd0c40c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.996+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/4186574326 >> 172.21.15.186:0/4186574326 conn(0x7fd0c40c8a20 msgr2=0x7fd0c40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:21.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.996+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/4186574326 shutdown_connections 2024-03-28T15:10:21.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.996+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/4186574326 wait complete. 2024-03-28T15:10:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.997+0000 7fd0cb8ec640 1 Processor -- start 2024-03-28T15:10:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.997+0000 7fd0cb8ec640 1 -- start start 2024-03-28T15:10:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0cb8ec640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0c40b1e30 0x7fd0c41e3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0cb8ec640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0c40c2d00 0x7fd0c4152980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0cb8ec640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0c4151990 0x7fd0c41e2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0cb8ec640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c4073dd0 con 0x7fd0c40b1e30 2024-03-28T15:10:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0cb8ec640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c40c1900 con 0x7fd0c4151990 2024-03-28T15:10:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0cb8ec640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd0c4073f50 con 0x7fd0c40c2d00 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c9688640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0c40b1e30 0x7fd0c41e3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c9688640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0c40b1e30 0x7fd0c41e3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54900/0 (socket says 172.21.15.186:54900) 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c9688640 1 -- 172.21.15.186:0/1005586682 learned_addr learned my addr 172.21.15.186:0/1005586682 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c8e87640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0c40c2d00 0x7fd0c4152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c9e89640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0c4151990 0x7fd0c41e2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c9688640 1 -- 172.21.15.186:0/1005586682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0c40c2d00 msgr2=0x7fd0c4152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c9688640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0c40c2d00 0x7fd0c4152980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c9688640 1 -- 172.21.15.186:0/1005586682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0c4151990 msgr2=0x7fd0c41e2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c9688640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0c4151990 0x7fd0c41e2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c9688640 1 -- 172.21.15.186:0/1005586682 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0c414f750 con 0x7fd0c40b1e30 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c8e87640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0c40c2d00 0x7fd0c4152980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c9e89640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0c4151990 0x7fd0c41e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0c9688640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0c40b1e30 0x7fd0c41e3bf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000c820 tx=0x7fd0c000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0b9ffb640 1 -- 172.21.15.186:0/1005586682 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0015070 con 0x7fd0c40b1e30 2024-03-28T15:10:22.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0b9ffb640 1 -- 172.21.15.186:0/1005586682 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0c0004240 con 0x7fd0c40b1e30 2024-03-28T15:10:22.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/1005586682 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0c41e2e30 con 0x7fd0c40b1e30 2024-03-28T15:10:22.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.998+0000 7fd0b9ffb640 1 -- 172.21.15.186:0/1005586682 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c000ce10 con 0x7fd0c40b1e30 2024-03-28T15:10:22.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:21.999+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/1005586682 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0c41e33f0 con 0x7fd0c40b1e30 2024-03-28T15:10:22.002 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.000+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/1005586682 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0c4069500 con 0x7fd0c40b1e30 2024-03-28T15:10:22.003 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.000+0000 7fd0b9ffb640 1 -- 172.21.15.186:0/1005586682 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd0c0007830 con 0x7fd0c40b1e30 2024-03-28T15:10:22.003 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.001+0000 7fd0b9ffb640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd0a806c560 0x7fd0a806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:22.003 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.001+0000 7fd0b9ffb640 1 -- 172.21.15.186:0/1005586682 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fd0c008b740 con 0x7fd0c40b1e30 2024-03-28T15:10:22.003 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.001+0000 7fd0c8e87640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd0a806c560 0x7fd0a806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:22.003 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.001+0000 7fd0c8e87640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd0a806c560 0x7fd0a806ea20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4036000 tx=0x7fd0b403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:22.005 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.005+0000 7fd0b9ffb640 1 -- 172.21.15.186:0/1005586682 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd0c0059bd0 con 0x7fd0c40b1e30 2024-03-28T15:10:22.221 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.221+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/1005586682 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0c40b1a00 con 0x7fd0c40b1e30 2024-03-28T15:10:22.222 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.221+0000 7fd0b9ffb640 1 -- 172.21.15.186:0/1005586682 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6019 (secure 0 0 0) 0x7fd0c00594d0 con 0x7fd0c40b1e30 2024-03-28T15:10:22.222 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:22.222 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:16.464120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:clientreplay","state_seq":23,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-28T15:10:22.222 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 33 2024-03-28T15:10:22.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/1005586682 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd0a806c560 msgr2=0x7fd0a806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:22.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd0a806c560 0x7fd0a806ea20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4036000 tx=0x7fd0b403b040 comp rx=0 tx=0).stop 2024-03-28T15:10:22.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/1005586682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0c40b1e30 msgr2=0x7fd0c41e3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:22.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0c40b1e30 0x7fd0c41e3bf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000c820 tx=0x7fd0c000cce0 comp rx=0 tx=0).stop 2024-03-28T15:10:22.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/1005586682 shutdown_connections 2024-03-28T15:10:22.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd0a806c560 0x7fd0a806ea20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:22.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0c4151990 0x7fd0c41e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:22.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0c40c2d00 0x7fd0c4152980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:22.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 --2- 172.21.15.186:0/1005586682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0c40b1e30 0x7fd0c41e3bf0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:22.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/1005586682 >> 172.21.15.186:0/1005586682 conn(0x7fd0c40c8a20 msgr2=0x7fd0c4147b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:22.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/1005586682 shutdown_connections 2024-03-28T15:10:22.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:22.223+0000 7fd0cb8ec640 1 -- 172.21.15.186:0/1005586682 wait complete. 2024-03-28T15:10:22.236 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 33, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T15:01:31.425299+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 33, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5278": { "addr": "172.21.15.191:6834/3646031339", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 3646031339, "type": "v2" }, { "addr": "172.21.15.191:6834", "nonce": 3646031339, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5278, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:clientreplay", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T15:10:16.464120+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5278 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/1278716016", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6833", "nonce": 1278716016, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1278716016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6835/2068761001", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2068761001, "type": "v2" }, { "addr": "172.21.15.186:6835", "nonce": 2068761001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2344737463", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6837", "nonce": 2344737463, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2344737463, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4284, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/1548903381", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 1548903381, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 1548903381, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T15:10:24.237 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:24.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.338+0000 7fd90518a640 1 Processor -- start 2024-03-28T15:10:24.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.339+0000 7fd90518a640 1 -- start start 2024-03-28T15:10:24.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.339+0000 7fd90518a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd90006f410 0x7fd9000c3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:24.343 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.339+0000 7fd90518a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd900072410 0x7fd9000c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:24.343 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.339+0000 7fd90518a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd90006eaf0 con 0x7fd9001571a0 2024-03-28T15:10:24.343 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.339+0000 7fd90518a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd9000c9070 con 0x7fd90006f410 2024-03-28T15:10:24.343 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.339+0000 7fd90518a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fd9000c84f0 con 0x7fd900072410 2024-03-28T15:10:24.343 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.339+0000 7fd8f7fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd90006f410 0x7fd9000c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:24.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.339+0000 7fd8fed9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd9001571a0 0x7fd90015b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:37602/0 (socket says 172.21.15.186:37602) 2024-03-28T15:10:24.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.339+0000 7fd8fed9d640 1 -- 172.21.15.186:0/4284144851 learned_addr learned my addr 172.21.15.186:0/4284144851 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:24.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.339+0000 7fd8ff59e640 1 --2- 172.21.15.186:0/4284144851 >> v2:172.21.15.191:3300/0 conn(0x7fd900072410 0x7fd9000c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:24.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.340+0000 7fd8f7fff640 1 -- 172.21.15.186:0/4284144851 >> v2:172.21.15.191:3300/0 conn(0x7fd900072410 msgr2=0x7fd9000c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:24.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.340+0000 7fd8f7fff640 1 --2- 172.21.15.186:0/4284144851 >> v2:172.21.15.191:3300/0 conn(0x7fd900072410 0x7fd9000c2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:24.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.340+0000 7fd8f7fff640 1 -- 172.21.15.186:0/4284144851 >> v1:172.21.15.186:6789/0 conn(0x7fd9001571a0 legacy=0x7fd90015b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:24.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.340+0000 7fd8f7fff640 1 -- 172.21.15.186:0/4284144851 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9000c87d0 con 0x7fd90006f410 2024-03-28T15:10:24.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.340+0000 7fd8fe59c640 1 -- 172.21.15.186:0/4284144851 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3391036253 0 0) 0x7fd8f0001110 con 0x7fd9001571a0 2024-03-28T15:10:24.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.340+0000 7fd8f7fff640 1 --2- 172.21.15.186:0/4284144851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90006f410 0x7fd9000c3790 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd9000cb030 tx=0x7fd8e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77da692b691d6d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:24.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.341+0000 7fd8fe59c640 1 -- 172.21.15.186:0/4284144851 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8034070 con 0x7fd90006f410 2024-03-28T15:10:24.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.341+0000 7fd8fe59c640 1 -- 172.21.15.186:0/4284144851 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e8030610 con 0x7fd90006f410 2024-03-28T15:10:24.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.341+0000 7fd8fe59c640 1 -- 172.21.15.186:0/4284144851 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80309e0 con 0x7fd90006f410 2024-03-28T15:10:24.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.341+0000 7fd90518a640 1 -- 172.21.15.186:0/4284144851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90006f410 msgr2=0x7fd9000c3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:24.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.341+0000 7fd90518a640 1 --2- 172.21.15.186:0/4284144851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90006f410 0x7fd9000c3790 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd9000cb030 tx=0x7fd8e802b960 comp rx=0 tx=0).stop 2024-03-28T15:10:24.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.341+0000 7fd90518a640 1 -- 172.21.15.186:0/4284144851 shutdown_connections 2024-03-28T15:10:24.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.341+0000 7fd90518a640 1 --2- 172.21.15.186:0/4284144851 >> v2:172.21.15.191:3300/0 conn(0x7fd900072410 0x7fd9000c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:24.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.341+0000 7fd90518a640 1 --2- 172.21.15.186:0/4284144851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90006f410 0x7fd9000c3790 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:24.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.341+0000 7fd90518a640 1 -- 172.21.15.186:0/4284144851 >> 172.21.15.186:0/4284144851 conn(0x7fd9000cd360 msgr2=0x7fd9000cd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:24.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.341+0000 7fd90518a640 1 -- 172.21.15.186:0/4284144851 shutdown_connections 2024-03-28T15:10:24.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.341+0000 7fd90518a640 1 -- 172.21.15.186:0/4284144851 wait complete. 2024-03-28T15:10:24.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.342+0000 7fd90518a640 1 Processor -- start 2024-03-28T15:10:24.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.342+0000 7fd90518a640 1 -- start start 2024-03-28T15:10:24.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd90518a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90006f410 0x7fd900151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:24.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd90518a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd900072410 0x7fd900151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:24.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd90518a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9001571a0 0x7fd900154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd90518a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd90015ad90 con 0x7fd90006f410 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd90518a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd9000c8d90 con 0x7fd9001571a0 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd90518a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fd90015b9c0 con 0x7fd900072410 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd8fed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90006f410 0x7fd900151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd8fed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90006f410 0x7fd900151890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54910/0 (socket says 172.21.15.186:54910) 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd8fed9d640 1 -- 172.21.15.186:0/913120884 learned_addr learned my addr 172.21.15.186:0/913120884 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd8f7fff640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd900072410 0x7fd900151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd8ff59e640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9001571a0 0x7fd900154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd8fed9d640 1 -- 172.21.15.186:0/913120884 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd900072410 msgr2=0x7fd900151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd8fed9d640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd900072410 0x7fd900151dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd8fed9d640 1 -- 172.21.15.186:0/913120884 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9001571a0 msgr2=0x7fd900154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd8fed9d640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9001571a0 0x7fd900154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.343+0000 7fd8fed9d640 1 -- 172.21.15.186:0/913120884 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd900155d20 con 0x7fd90006f410 2024-03-28T15:10:24.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.344+0000 7fd8fed9d640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90006f410 0x7fd900151890 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000c430 tx=0x7fd8f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:24.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.344+0000 7fd8f77fe640 1 -- 172.21.15.186:0/913120884 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f00074e0 con 0x7fd90006f410 2024-03-28T15:10:24.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.344+0000 7fd8f77fe640 1 -- 172.21.15.186:0/913120884 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f00047e0 con 0x7fd90006f410 2024-03-28T15:10:24.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.344+0000 7fd90518a640 1 -- 172.21.15.186:0/913120884 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd900159d10 con 0x7fd90006f410 2024-03-28T15:10:24.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.344+0000 7fd8f77fe640 1 -- 172.21.15.186:0/913120884 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0004bd0 con 0x7fd90006f410 2024-03-28T15:10:24.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.344+0000 7fd90518a640 1 -- 172.21.15.186:0/913120884 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd90015a2d0 con 0x7fd90006f410 2024-03-28T15:10:24.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.345+0000 7fd90518a640 1 -- 172.21.15.186:0/913120884 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd900068f00 con 0x7fd90006f410 2024-03-28T15:10:24.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.345+0000 7fd8f77fe640 1 -- 172.21.15.186:0/913120884 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd8f0011070 con 0x7fd90006f410 2024-03-28T15:10:24.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.346+0000 7fd8f77fe640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8e006c560 0x7fd8e006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:24.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.346+0000 7fd8f77fe640 1 -- 172.21.15.186:0/913120884 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fd8f008b4c0 con 0x7fd90006f410 2024-03-28T15:10:24.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.346+0000 7fd8f7fff640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8e006c560 0x7fd8e006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:24.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.346+0000 7fd8f7fff640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8e006c560 0x7fd8e006ea20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8036000 tx=0x7fd8e8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:24.352 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.350+0000 7fd8f77fe640 1 -- 172.21.15.186:0/913120884 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fd900068f00 con 0x7fd90006f410 2024-03-28T15:10:24.561 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.560+0000 7fd90518a640 1 -- 172.21.15.186:0/913120884 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9000686d0 con 0x7fd90006f410 2024-03-28T15:10:24.561 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.561+0000 7fd8f77fe640 1 -- 172.21.15.186:0/913120884 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6020 (secure 0 0 0) 0x7fd90006eaf0 con 0x7fd90006f410 2024-03-28T15:10:24.562 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:24.562 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:24.324220+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":25,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5278,"qdb_cluster":[5278]},"id":1}]} 2024-03-28T15:10:24.562 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 34 2024-03-28T15:10:24.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 -- 172.21.15.186:0/913120884 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8e006c560 msgr2=0x7fd8e006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:24.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8e006c560 0x7fd8e006ea20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8036000 tx=0x7fd8e8008030 comp rx=0 tx=0).stop 2024-03-28T15:10:24.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 -- 172.21.15.186:0/913120884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90006f410 msgr2=0x7fd900151890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:24.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90006f410 0x7fd900151890 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000c430 tx=0x7fd8f000c8f0 comp rx=0 tx=0).stop 2024-03-28T15:10:24.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 -- 172.21.15.186:0/913120884 shutdown_connections 2024-03-28T15:10:24.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fd8e006c560 0x7fd8e006ea20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:24.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9001571a0 0x7fd900154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:24.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd900072410 0x7fd900151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:24.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 --2- 172.21.15.186:0/913120884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90006f410 0x7fd900151890 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:24.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 -- 172.21.15.186:0/913120884 >> 172.21.15.186:0/913120884 conn(0x7fd9000cd360 msgr2=0x7fd900148620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:24.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 -- 172.21.15.186:0/913120884 shutdown_connections 2024-03-28T15:10:24.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:24.563+0000 7fd90518a640 1 -- 172.21.15.186:0/913120884 wait complete. 2024-03-28T15:10:24.575 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5137 2024-03-28T15:10:24.575 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.f 2024-03-28T15:10:27.633 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:10:27.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:10:30.877 INFO:tasks.ceph.mds.b.smithi191.stderr:2024-03-28T15:10:30.876+0000 7ffba1cb8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T15:10:32.576 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-03-28T15:10:32.576 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-28T15:10:32.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-03-28T15:10:32.578 INFO:tasks.ceph.mds.f:Started 2024-03-28T15:10:32.579 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:32.667 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2024-03-28T15:10:32.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.694+0000 7fb6c73d0640 1 Processor -- start 2024-03-28T15:10:32.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.694+0000 7fb6c73d0640 1 -- start start 2024-03-28T15:10:32.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.695+0000 7fb6c73d0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb6c00c5380 0x7fb6c00cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:32.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.695+0000 7fb6c73d0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb6c00cb390 0x7fb6c0165220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:32.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.695+0000 7fb6c73d0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6c00c6f90 con 0x7fb6c00caa70 2024-03-28T15:10:32.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.695+0000 7fb6c73d0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb6c0071a80 con 0x7fb6c00c5380 2024-03-28T15:10:32.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.695+0000 7fb6c73d0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fb6c00c6030 con 0x7fb6c00cb390 2024-03-28T15:10:32.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.695+0000 7fb6c496b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb6c00c5380 0x7fb6c00cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:32.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.695+0000 7fb6c496b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb6c00c5380 0x7fb6c00cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52724/0 (socket says 172.21.15.186:52724) 2024-03-28T15:10:32.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.695+0000 7fb6c496b640 1 -- 172.21.15.186:0/2050912437 learned_addr learned my addr 172.21.15.186:0/2050912437 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:32.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.695+0000 7fb6c596d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb6c00cb390 0x7fb6c0165220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:32.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.696+0000 7fb6c496b640 1 -- 172.21.15.186:0/2050912437 >> v2:172.21.15.191:3300/0 conn(0x7fb6c00cb390 msgr2=0x7fb6c0165220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:32.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.696+0000 7fb6c496b640 1 --2- 172.21.15.186:0/2050912437 >> v2:172.21.15.191:3300/0 conn(0x7fb6c00cb390 0x7fb6c0165220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:32.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.696+0000 7fb6c496b640 1 -- 172.21.15.186:0/2050912437 >> v1:172.21.15.191:6789/0 conn(0x7fb6c00caa70 legacy=0x7fb6c00c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:32.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.696+0000 7fb6c496b640 1 -- 172.21.15.186:0/2050912437 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c00cc970 con 0x7fb6c00c5380 2024-03-28T15:10:32.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.697+0000 7fb6c496b640 1 --2- 172.21.15.186:0/2050912437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6c00c5380 0x7fb6c00cae50 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb6c008d380 tx=0x7fb6b002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a82298c04e365c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:32.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.697+0000 7fb6b7fff640 1 -- 172.21.15.186:0/2050912437 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0034070 con 0x7fb6c00c5380 2024-03-28T15:10:32.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.697+0000 7fb6b7fff640 1 -- 172.21.15.186:0/2050912437 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6b0030860 con 0x7fb6c00c5380 2024-03-28T15:10:32.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.697+0000 7fb6b7fff640 1 -- 172.21.15.186:0/2050912437 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0030c30 con 0x7fb6c00c5380 2024-03-28T15:10:32.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.698+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2050912437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6c00c5380 msgr2=0x7fb6c00cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:32.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.698+0000 7fb6c73d0640 1 --2- 172.21.15.186:0/2050912437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6c00c5380 0x7fb6c00cae50 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb6c008d380 tx=0x7fb6b002b9a0 comp rx=0 tx=0).stop 2024-03-28T15:10:32.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.698+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2050912437 shutdown_connections 2024-03-28T15:10:32.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.698+0000 7fb6c73d0640 1 --2- 172.21.15.186:0/2050912437 >> v2:172.21.15.191:3300/0 conn(0x7fb6c00cb390 0x7fb6c0165220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:32.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.698+0000 7fb6c73d0640 1 --2- 172.21.15.186:0/2050912437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6c00c5380 0x7fb6c00cae50 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:32.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.698+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2050912437 >> 172.21.15.186:0/2050912437 conn(0x7fb6c00d00f0 msgr2=0x7fb6c00d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:32.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.698+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2050912437 shutdown_connections 2024-03-28T15:10:32.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.698+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2050912437 wait complete. 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.699+0000 7fb6c73d0640 1 Processor -- start 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.699+0000 7fb6c73d0640 1 -- start start 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c73d0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6c00c5380 0x7fb6c014f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c73d0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6c00caa70 0x7fb6c00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c73d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6c00cb390 0x7fb6c01f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c73d0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c00c5cc0 con 0x7fb6c00caa70 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c73d0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c015b530 con 0x7fb6c00c5380 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c73d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fb6c00cb840 con 0x7fb6c00cb390 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c496b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6c00caa70 0x7fb6c00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c516c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6c00c5380 0x7fb6c014f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c496b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6c00caa70 0x7fb6c00b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52730/0 (socket says 172.21.15.186:52730) 2024-03-28T15:10:32.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c496b640 1 -- 172.21.15.186:0/2618971956 learned_addr learned my addr 172.21.15.186:0/2618971956 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c596d640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6c00cb390 0x7fb6c01f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c496b640 1 -- 172.21.15.186:0/2618971956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6c00cb390 msgr2=0x7fb6c01f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c496b640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6c00cb390 0x7fb6c01f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c496b640 1 -- 172.21.15.186:0/2618971956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6c00c5380 msgr2=0x7fb6c014f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c496b640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6c00c5380 0x7fb6c014f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c496b640 1 -- 172.21.15.186:0/2618971956 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c01e3640 con 0x7fb6c00caa70 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.700+0000 7fb6c496b640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6c00caa70 0x7fb6c00b5700 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00097c0 tx=0x7fb6b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.701+0000 7fb6b5ffb640 1 -- 172.21.15.186:0/2618971956 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0048070 con 0x7fb6c00caa70 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.701+0000 7fb6b5ffb640 1 -- 172.21.15.186:0/2618971956 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6b0030660 con 0x7fb6c00caa70 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.701+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2618971956 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c01e3360 con 0x7fb6c00caa70 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.701+0000 7fb6b5ffb640 1 -- 172.21.15.186:0/2618971956 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0034040 con 0x7fb6c00caa70 2024-03-28T15:10:32.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.701+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2618971956 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c01e2db0 con 0x7fb6c00caa70 2024-03-28T15:10:32.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.702+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2618971956 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6c00686d0 con 0x7fb6c00caa70 2024-03-28T15:10:32.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.702+0000 7fb6b5ffb640 1 -- 172.21.15.186:0/2618971956 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb6b004b070 con 0x7fb6c00caa70 2024-03-28T15:10:32.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.703+0000 7fb6b5ffb640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb6a406c530 0x7fb6a406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:32.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.703+0000 7fb6b5ffb640 1 -- 172.21.15.186:0/2618971956 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fb6b00ad670 con 0x7fb6c00caa70 2024-03-28T15:10:32.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.703+0000 7fb6c516c640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb6a406c530 0x7fb6a406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:32.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.703+0000 7fb6c516c640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb6a406c530 0x7fb6a406e9f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc001780 tx=0x7fb6bc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:32.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.709+0000 7fb6b5ffb640 1 -- 172.21.15.186:0/2618971956 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fb6b007b4b0 con 0x7fb6c00caa70 2024-03-28T15:10:32.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.964+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2618971956 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6c014efd0 con 0x7fb6c00caa70 2024-03-28T15:10:32.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.964+0000 7fb6b5ffb640 1 -- 172.21.15.186:0/2618971956 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6020 (secure 0 0 0) 0x7fb6b007f7b0 con 0x7fb6c00caa70 2024-03-28T15:10:32.965 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:32.965 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:24.324220+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":25,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5278,"qdb_cluster":[5278]},"id":1}]} 2024-03-28T15:10:32.965 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 34 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2618971956 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb6a406c530 msgr2=0x7fb6a406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb6a406c530 0x7fb6a406e9f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc001780 tx=0x7fb6bc009040 comp rx=0 tx=0).stop 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2618971956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6c00caa70 msgr2=0x7fb6c00b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6c00caa70 0x7fb6c00b5700 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00097c0 tx=0x7fb6b0004730 comp rx=0 tx=0).stop 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2618971956 shutdown_connections 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fb6a406c530 0x7fb6a406e9f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6c00cb390 0x7fb6c01f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6c00caa70 0x7fb6c00b5700 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 --2- 172.21.15.186:0/2618971956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6c00c5380 0x7fb6c014f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2618971956 >> 172.21.15.186:0/2618971956 conn(0x7fb6c00d00f0 msgr2=0x7fb6c00cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2618971956 shutdown_connections 2024-03-28T15:10:32.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:32.967+0000 7fb6c73d0640 1 -- 172.21.15.186:0/2618971956 wait complete. 2024-03-28T15:10:32.980 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2024-03-28T15:10:33.802 INFO:tasks.workunit.client.0.smithi186.stderr:Updating files: 3% (423/12435) Updating files: 4% (498/12435) Updating files: 5% (622/12435) Updating files: 6% (747/12435) Updating files: 6% (788/12435) Updating files: 7% (871/12435) Updating files: 8% (995/12435) Updating files: 8% (1061/12435) Updating files: 9% (1120/12435) Updating files: 10% (1244/12435) Updating files: 11% (1368/12435) Updating files: 11% (1427/12435) Updating files: 12% (1493/12435) Updating files: 13% (1617/12435) Updating files: 13% (1666/12435) Updating files: 14% (1741/12435) Updating files: 15% (1866/12435) Updating files: 15% (1966/12435) Updating files: 16% (1990/12435) Updating files: 17% (2114/12435) Updating files: 17% (2171/12435) Updating files: 18% (2239/12435) Updating files: 19% (2363/12435) Updating files: 19% (2432/12435) Updating files: 20% (2487/12435) Updating files: 21% (2612/12435) Updating files: 21% (2699/12435) Updating files: 22% (2736/12435) Updating files: 23% (2861/12435) Updating files: 23% (2893/12435) Updating files: 24% (2985/12435) Updating files: 25% (3109/12435) Updating files: 25% (3211/12435) Updating files: 26% (3234/12435) Updating files: 27% (3358/12435) Updating files: 28% (3482/12435) Updating files: 28% (3561/12435) Updating files: 29% (3607/12435) Updating files: 30% (3731/12435) Updating files: 30% (3743/12435) Updating files: 31% (3855/12435) Updating files: 32% (3980/12435) Updating files: 32% (4052/12435) Updating files: 33% (4104/12435) Updating files: 34% (4228/12435) Updating files: 35% (4353/12435) Updating files: 35% (4376/12435) Updating files: 36% (4477/12435) Updating files: 37% (4601/12435) Updating files: 37% (4661/12435) Updating files: 38% (4726/12435) Updating files: 39% (4850/12435) Updating files: 39% (4969/12435) Updating files: 40% (4974/12435) Updating files: 41% (5099/12435) Updating files: 42% (5223/12435) Updating files: 42% (5259/12435) Updating files: 43% (5348/12435) Updating files: 44% (5472/12435) Updating files: 44% (5524/12435) Updating files: 45% (5596/12435) Updating files: 46% (5721/12435) Updating files: 46% (5818/12435) Updating files: 47% (5845/12435) Updating files: 48% (5969/12435) Updating files: 49% (6094/12435) Updating files: 49% (6177/12435) Updating files: 50% (6218/12435) Updating files: 51% (6342/12435) Updating files: 52% (6467/12435) Updating files: 52% (6481/12435) Updating files: 53% (6591/12435) Updating files: 54% (6715/12435) Updating files: 54% (6792/12435) Updating files: 55% (6840/12435) Updating files: 56% (6964/12435) Updating files: 57% (7088/12435) Updating files: 57% (7142/12435) Updating files: 58% (7213/12435) Updating files: 59% (7337/12435) Updating files: 60% (7461/12435) Updating files: 60% (7467/12435) Updating files: 61% (7586/12435) Updating files: 62% (7710/12435) Updating files: 62% (7809/12435) Updating files: 63% (7835/12435) Updating files: 64% (7959/12435) Updating files: 65% (8083/12435) Updating files: 65% (8102/12435) Updating files: 66% (8208/12435) Updating files: 67% (8332/12435) Updating files: 67% (8334/12435) Updating files: 68% (8456/12435) Updating files: 68% (8525/12435) Updating files: 69% (8581/12435) Updating files: 69% (8687/12435) Updating files: 70% (8705/12435) Updating files: 71% (8829/12435) Updating files: 71% (8850/12435) Updating files: 72% (8954/12435) Updating files: 72% (9034/12435) Updating files: 73% (9078/12435) Updating files: 74% (9202/12435) Updating files: 74% (9235/12435) Updating files: 75% (9327/12435) Updating files: 76% (9451/12435) Updating files: 76% (9454/12435) Updating files: 77% (9575/12435) Updating files: 77% (9699/12435) Updating files: 78% (9700/12435) Updating files: 79% (9824/12435) Updating files: 79% (9845/12435) Updating files: 79% (9928/12435) Updating files: 80% (9948/12435) Updating files: 81% (10073/12435) Updating files: 81% (10187/12435) Updating files: 82% (10197/12435) Updating files: 83% (10322/12435) Updating files: 84% (10446/12435) Updating files: 84% (10523/12435) Updating files: 85% (10570/12435) Updating files: 85% (10665/12435) Updating files: 86% (10695/12435) Updating files: 87% (10819/12435) Updating files: 88% (10943/12435) Updating files: 88% (10966/12435) Updating files: 88% (10974/12435) Updating files: 89% (11068/12435) Updating files: 90% (11192/12435) Updating files: 90% (11291/12435) Updating files: 91% (11316/12435) Updating files: 92% (11441/12435) Updating files: 93% (11565/12435) Updating files: 93% (11666/12435) Updating files: 94% (11689/12435) Updating files: 95% (11814/12435) Updating files: 96% (11938/12435) Updating files: 96% (11977/12435) Updating files: 97% (12062/12435) Updating files: 98% (12187/12435) Updating files: 99% (12311/12435) Updating files: 99% (12350/12435) Updating files: 100% (12435/12435) Updating files: 100% (12435/12435), done. 2024-03-28T15:10:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 128 2024-03-28T15:10:33.838 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2024-03-28T15:10:33.838 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-28T15:10:34.115 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi186 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-03-28T15:10:34.364 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=49afb54454524bd3a776f2646495f48c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi186 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-03-28T15:10:34.367 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-03-28T15:10:34.377 INFO:tasks.mds_thrash:joining mds_thrasher 2024-03-28T15:10:34.981 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:10:35.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.084+0000 7fa1fb5b5640 1 Processor -- start 2024-03-28T15:10:35.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.084+0000 7fa1fb5b5640 1 -- start start 2024-03-28T15:10:35.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1fb5b5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa1f4151990 0x7fa1f40c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1fb5b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1f40b1e30 0x7fa1f40c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1fb5b5640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa1f40c2d00 0x7fa1f40712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1fb5b5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa1f40c1da0 con 0x7fa1f4151990 2024-03-28T15:10:35.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1fb5b5640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fa1f406caa0 con 0x7fa1f40c2d00 2024-03-28T15:10:35.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1fb5b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f406cd80 con 0x7fa1f40b1e30 2024-03-28T15:10:35.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1f9351640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa1f4151990 0x7fa1f40c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1f9351640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa1f4151990 0x7fa1f40c2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52734/0 (socket says 172.21.15.186:52734) 2024-03-28T15:10:35.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1f9b52640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa1f40c2d00 0x7fa1f40712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1f8b50640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1f40b1e30 0x7fa1f40c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1f9351640 1 -- 172.21.15.186:0/514012415 learned_addr learned my addr 172.21.15.186:0/514012415 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:35.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1f9351640 1 -- 172.21.15.186:0/514012415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1f40b1e30 msgr2=0x7fa1f40c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:35.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1f9351640 1 --2- 172.21.15.186:0/514012415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1f40b1e30 0x7fa1f40c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.085+0000 7fa1f9351640 1 -- 172.21.15.186:0/514012415 >> v2:172.21.15.191:3300/0 conn(0x7fa1f40c2d00 msgr2=0x7fa1f40712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:35.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.086+0000 7fa1f9351640 1 --2- 172.21.15.186:0/514012415 >> v2:172.21.15.191:3300/0 conn(0x7fa1f40c2d00 0x7fa1f40712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.086+0000 7fa1f9351640 1 -- 172.21.15.186:0/514012415 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f40c1340 con 0x7fa1f4151990 2024-03-28T15:10:35.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.086+0000 7fa1f9351640 1 --2- 172.21.15.186:0/514012415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1f4151990 0x7fa1f40c2230 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40ccfc0 tx=0x7fa1f002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f79efda3efdefaa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:35.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.086+0000 7fa1ebfff640 1 -- 172.21.15.186:0/514012415 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f002cb00 con 0x7fa1f4151990 2024-03-28T15:10:35.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.086+0000 7fa1ebfff640 1 -- 172.21.15.186:0/514012415 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f002cca0 con 0x7fa1f4151990 2024-03-28T15:10:35.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.086+0000 7fa1ebfff640 1 -- 172.21.15.186:0/514012415 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0034610 con 0x7fa1f4151990 2024-03-28T15:10:35.092 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.087+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/514012415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1f4151990 msgr2=0x7fa1f40c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.092 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.087+0000 7fa1fb5b5640 1 --2- 172.21.15.186:0/514012415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1f4151990 0x7fa1f40c2230 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40ccfc0 tx=0x7fa1f002bf60 comp rx=0 tx=0).stop 2024-03-28T15:10:35.092 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.087+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/514012415 shutdown_connections 2024-03-28T15:10:35.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.087+0000 7fa1fb5b5640 1 --2- 172.21.15.186:0/514012415 >> v2:172.21.15.191:3300/0 conn(0x7fa1f40c2d00 0x7fa1f40712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.087+0000 7fa1fb5b5640 1 --2- 172.21.15.186:0/514012415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1f40b1e30 0x7fa1f40c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.087+0000 7fa1fb5b5640 1 --2- 172.21.15.186:0/514012415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1f4151990 0x7fa1f40c2230 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.087+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/514012415 >> 172.21.15.186:0/514012415 conn(0x7fa1f40c8a20 msgr2=0x7fa1f40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:35.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.087+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/514012415 shutdown_connections 2024-03-28T15:10:35.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.087+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/514012415 wait complete. 2024-03-28T15:10:35.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.088+0000 7fa1fb5b5640 1 Processor -- start 2024-03-28T15:10:35.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.088+0000 7fa1fb5b5640 1 -- start start 2024-03-28T15:10:35.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1fb5b5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1f40b1e30 0x7fa1f4147b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1fb5b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1f40c2d00 0x7fa1f40c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1fb5b5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1f4151990 0x7fa1f41e2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1fb5b5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f4073dd0 con 0x7fa1f40b1e30 2024-03-28T15:10:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1fb5b5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f40c1900 con 0x7fa1f4151990 2024-03-28T15:10:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1fb5b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f4073f50 con 0x7fa1f40c2d00 2024-03-28T15:10:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1f9351640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1f40b1e30 0x7fa1f4147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1f9351640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1f40b1e30 0x7fa1f4147b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52750/0 (socket says 172.21.15.186:52750) 2024-03-28T15:10:35.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1f9351640 1 -- 172.21.15.186:0/2444918783 learned_addr learned my addr 172.21.15.186:0/2444918783 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1f8b50640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1f40c2d00 0x7fa1f40c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1f9b52640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1f4151990 0x7fa1f41e2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1f9351640 1 -- 172.21.15.186:0/2444918783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1f40c2d00 msgr2=0x7fa1f40c5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1f9351640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1f40c2d00 0x7fa1f40c5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1f9351640 1 -- 172.21.15.186:0/2444918783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1f4151990 msgr2=0x7fa1f41e2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1f9351640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1f4151990 0x7fa1f41e2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1f9351640 1 -- 172.21.15.186:0/2444918783 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f414eaa0 con 0x7fa1f40b1e30 2024-03-28T15:10:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1f9351640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1f40b1e30 0x7fa1f4147b40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0036040 tx=0x7fa1f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:35.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.089+0000 7fa1e9ffb640 1 -- 172.21.15.186:0/2444918783 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0043020 con 0x7fa1f40b1e30 2024-03-28T15:10:35.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.090+0000 7fa1e9ffb640 1 -- 172.21.15.186:0/2444918783 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f002ce50 con 0x7fa1f40b1e30 2024-03-28T15:10:35.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.090+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/2444918783 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f41efd00 con 0x7fa1f40b1e30 2024-03-28T15:10:35.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.090+0000 7fa1e9ffb640 1 -- 172.21.15.186:0/2444918783 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0009510 con 0x7fa1f40b1e30 2024-03-28T15:10:35.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.090+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/2444918783 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f41f02c0 con 0x7fa1f40b1e30 2024-03-28T15:10:35.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.091+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/2444918783 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1f414f4e0 con 0x7fa1f40b1e30 2024-03-28T15:10:35.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.091+0000 7fa1e9ffb640 1 -- 172.21.15.186:0/2444918783 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa1f0005d10 con 0x7fa1f40b1e30 2024-03-28T15:10:35.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.092+0000 7fa1e9ffb640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa1d806c530 0x7fa1d806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.092+0000 7fa1e9ffb640 1 -- 172.21.15.186:0/2444918783 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fa1f00adfe0 con 0x7fa1f40b1e30 2024-03-28T15:10:35.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.092+0000 7fa1f8b50640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa1d806c530 0x7fa1d806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.092+0000 7fa1f8b50640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa1d806c530 0x7fa1d806e9f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa1f415ba10 tx=0x7fa1e4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:35.097 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.096+0000 7fa1e9ffb640 1 -- 172.21.15.186:0/2444918783 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa1f414f4e0 con 0x7fa1f40b1e30 2024-03-28T15:10:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.309+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/2444918783 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1f4069500 con 0x7fa1f40b1e30 2024-03-28T15:10:35.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.310+0000 7fa1e9ffb640 1 -- 172.21.15.186:0/2444918783 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6845 (secure 0 0 0) 0x7fa1f007bee0 con 0x7fa1f40b1e30 2024-03-28T15:10:35.312 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:35.312 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5407,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/249273897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":249273897},{"type":"v1","addr":"172.21.15.191:6837","nonce":249273897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:24.324220+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":25,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5278,"qdb_cluster":[5278]},"id":1}]} 2024-03-28T15:10:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 35 2024-03-28T15:10:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/2444918783 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa1d806c530 msgr2=0x7fa1d806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa1d806c530 0x7fa1d806e9f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa1f415ba10 tx=0x7fa1e4009190 comp rx=0 tx=0).stop 2024-03-28T15:10:35.315 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/2444918783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1f40b1e30 msgr2=0x7fa1f4147b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.315 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1f40b1e30 0x7fa1f4147b40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0036040 tx=0x7fa1f0004a40 comp rx=0 tx=0).stop 2024-03-28T15:10:35.315 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/2444918783 shutdown_connections 2024-03-28T15:10:35.315 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa1d806c530 0x7fa1d806e9f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.316 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1f4151990 0x7fa1f41e2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.316 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1f40c2d00 0x7fa1f40c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.316 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 --2- 172.21.15.186:0/2444918783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1f40b1e30 0x7fa1f4147b40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.316 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/2444918783 >> 172.21.15.186:0/2444918783 conn(0x7fa1f40c8a20 msgr2=0x7fa1f406d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:35.317 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/2444918783 shutdown_connections 2024-03-28T15:10:35.317 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.312+0000 7fa1fb5b5640 1 -- 172.21.15.186:0/2444918783 wait complete. 2024-03-28T15:10:35.324 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-03-28T15:10:35.324 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2024-03-28T15:10:35.324 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2024-03-28T15:10:35.324 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 5 2024-03-28T15:10:35.325 INFO:tasks.mds_thrash:done joining 2024-03-28T15:10:35.325 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-28T15:10:35.335 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-28T15:10:35.335 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:10:35.335 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T15:10:35.351 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:10:35.351 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T15:10:35.407 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:10:35.548 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.548+0000 7fa26115e640 1 Processor -- start 2024-03-28T15:10:35.549 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.548+0000 7fa26115e640 1 -- start start 2024-03-28T15:10:35.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa26115e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25c0c57e0 0x7fa25c0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa26115e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa25c0c6e00 con 0x7fa25c0c53c0 2024-03-28T15:10:35.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa26115e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa25c15b080 con 0x7fa25c154bd0 2024-03-28T15:10:35.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa26115e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa25c0c5ca0 con 0x7fa25c0c57e0 2024-03-28T15:10:35.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa253fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fa25c0c53c0 0x7fa25c0c66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:33458/0 (socket says 172.21.15.186:33458) 2024-03-28T15:10:35.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa253fff640 1 -- 172.21.15.186:0/927019514 learned_addr learned my addr 172.21.15.186:0/927019514 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:35.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa25b59e640 1 --2- 172.21.15.186:0/927019514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25c0c57e0 0x7fa25c0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa25a59c640 1 -- 172.21.15.186:0/927019514 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2461503660 0 0) 0x7fa25c0c6e00 con 0x7fa25c0c53c0 2024-03-28T15:10:35.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa25a59c640 1 -- 172.21.15.186:0/927019514 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa248003520 con 0x7fa25c0c53c0 2024-03-28T15:10:35.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa25a59c640 1 -- 172.21.15.186:0/927019514 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 464256380 0 0) 0x7fa25c15b080 con 0x7fa25c154bd0 2024-03-28T15:10:35.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa25a59c640 1 -- 172.21.15.186:0/927019514 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa25c0c6e00 con 0x7fa25c154bd0 2024-03-28T15:10:35.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa25b59e640 1 -- 172.21.15.186:0/927019514 >> v1:172.21.15.191:6789/0 conn(0x7fa25c154bd0 legacy=0x7fa25c15a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa25b59e640 1 -- 172.21.15.186:0/927019514 >> v1:172.21.15.186:6789/0 conn(0x7fa25c0c53c0 legacy=0x7fa25c0c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.549+0000 7fa25b59e640 1 -- 172.21.15.186:0/927019514 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25c15b360 con 0x7fa25c0c57e0 2024-03-28T15:10:35.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.550+0000 7fa25b59e640 1 --2- 172.21.15.186:0/927019514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25c0c57e0 0x7fa25c0cb040 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa254009dc0 tx=0x7fa25402bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0ecda945853ad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:35.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.550+0000 7fa25a59c640 1 -- 172.21.15.186:0/927019514 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254039070 con 0x7fa25c0c57e0 2024-03-28T15:10:35.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.550+0000 7fa25a59c640 1 -- 172.21.15.186:0/927019514 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa25402cd50 con 0x7fa25c0c57e0 2024-03-28T15:10:35.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.550+0000 7fa25a59c640 1 -- 172.21.15.186:0/927019514 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254034640 con 0x7fa25c0c57e0 2024-03-28T15:10:35.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.550+0000 7fa26115e640 1 -- 172.21.15.186:0/927019514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25c0c57e0 msgr2=0x7fa25c0cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.550+0000 7fa26115e640 1 --2- 172.21.15.186:0/927019514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25c0c57e0 0x7fa25c0cb040 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa254009dc0 tx=0x7fa25402bfa0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.551+0000 7fa26115e640 1 -- 172.21.15.186:0/927019514 shutdown_connections 2024-03-28T15:10:35.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.551+0000 7fa26115e640 1 --2- 172.21.15.186:0/927019514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25c0c57e0 0x7fa25c0cb040 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.551+0000 7fa26115e640 1 -- 172.21.15.186:0/927019514 >> 172.21.15.186:0/927019514 conn(0x7fa25c0731f0 msgr2=0x7fa25c073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:35.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.551+0000 7fa26115e640 1 -- 172.21.15.186:0/927019514 shutdown_connections 2024-03-28T15:10:35.556 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.551+0000 7fa26115e640 1 -- 172.21.15.186:0/927019514 wait complete. 2024-03-28T15:10:35.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.552+0000 7fa26115e640 1 Processor -- start 2024-03-28T15:10:35.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.552+0000 7fa26115e640 1 -- start start 2024-03-28T15:10:35.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.552+0000 7fa26115e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa25c0c53c0 0x7fa25c1f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.552+0000 7fa26115e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25c0c57e0 0x7fa25c15c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.552+0000 7fa26115e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa25c154bd0 0x7fa25c153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa26115e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa25c0c61d0 con 0x7fa25c154bd0 2024-03-28T15:10:35.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa26115e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa25c0c5ef0 con 0x7fa25c0c53c0 2024-03-28T15:10:35.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa26115e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fa25c0cc390 con 0x7fa25c0c57e0 2024-03-28T15:10:35.557 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa25b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa25c154bd0 0x7fa25c153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa25b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa25c154bd0 0x7fa25c153440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52758/0 (socket says 172.21.15.186:52758) 2024-03-28T15:10:35.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa25b59e640 1 -- 172.21.15.186:0/2770763188 learned_addr learned my addr 172.21.15.186:0/2770763188 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:35.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa25ad9d640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa25c0c53c0 0x7fa25c1f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa253fff640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25c0c57e0 0x7fa25c15c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa25b59e640 1 -- 172.21.15.186:0/2770763188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25c0c57e0 msgr2=0x7fa25c15c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa25b59e640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25c0c57e0 0x7fa25c15c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa25b59e640 1 -- 172.21.15.186:0/2770763188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa25c0c53c0 msgr2=0x7fa25c1f0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa25b59e640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa25c0c53c0 0x7fa25c1f0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa25b59e640 1 -- 172.21.15.186:0/2770763188 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25c1f3ba0 con 0x7fa25c154bd0 2024-03-28T15:10:35.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa25b59e640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa25c154bd0 0x7fa25c153440 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa254009ee0 tx=0x7fa254004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:35.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa2537fe640 1 -- 172.21.15.186:0/2770763188 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254039050 con 0x7fa25c154bd0 2024-03-28T15:10:35.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa2537fe640 1 -- 172.21.15.186:0/2770763188 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa254045070 con 0x7fa25c154bd0 2024-03-28T15:10:35.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa2537fe640 1 -- 172.21.15.186:0/2770763188 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254009450 con 0x7fa25c154bd0 2024-03-28T15:10:35.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa26115e640 1 -- 172.21.15.186:0/2770763188 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25c1f3e60 con 0x7fa25c154bd0 2024-03-28T15:10:35.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.553+0000 7fa26115e640 1 -- 172.21.15.186:0/2770763188 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa25c1f4420 con 0x7fa25c154bd0 2024-03-28T15:10:35.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.554+0000 7fa26115e640 1 -- 172.21.15.186:0/2770763188 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa25c069500 con 0x7fa25c154bd0 2024-03-28T15:10:35.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.555+0000 7fa2537fe640 1 -- 172.21.15.186:0/2770763188 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa25403e850 con 0x7fa25c154bd0 2024-03-28T15:10:35.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.555+0000 7fa2537fe640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa23c06c530 0x7fa23c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.555+0000 7fa2537fe640 1 -- 172.21.15.186:0/2770763188 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fa2540ae5f0 con 0x7fa25c154bd0 2024-03-28T15:10:35.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.555+0000 7fa25ad9d640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa23c06c530 0x7fa23c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.555+0000 7fa25ad9d640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa23c06c530 0x7fa23c06e9f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa24c007fd0 tx=0x7fa24c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:35.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.559+0000 7fa2537fe640 1 -- 172.21.15.186:0/2770763188 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fa25407c580 con 0x7fa25c154bd0 2024-03-28T15:10:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.735+0000 7fa26115e640 1 -- 172.21.15.186:0/2770763188 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa25c14b380 con 0x7fa25c154bd0 2024-03-28T15:10:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.737+0000 7fa2537fe640 1 -- 172.21.15.186:0/2770763188 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+17572 (secure 0 0 0) 0x7fa254080880 con 0x7fa25c154bd0 2024-03-28T15:10:35.737 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:35.738 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":37,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:09:56.237674+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.191:6834/1992706523":"2024-03-29T15:08:31.220829+0000","172.21.15.191:6837/3037586083":"2024-03-29T15:09:56.237602+0000","172.21.15.191:6835/3037586083":"2024-03-29T15:09:56.237602+0000","172.21.15.191:6837/3828930138":"2024-03-29T15:06:06.202259+0000","172.21.15.191:6835/3828930138":"2024-03-29T15:06:06.202259+0000","172.21.15.191:6832/1992706523":"2024-03-29T15:08:31.220829+0000","172.21.15.191:6834/185536353":"2024-03-29T15:03:46.182756+0000","172.21.15.191:6832/185536353":"2024-03-29T15:03:46.182756+0000","172.21.15.191:6837/2813406930":"2024-03-29T15:02:11.169402+0000","172.21.15.191:6835/2813406930":"2024-03-29T15:02:11.169402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:10:35.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 -- 172.21.15.186:0/2770763188 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa23c06c530 msgr2=0x7fa23c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa23c06c530 0x7fa23c06e9f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa24c007fd0 tx=0x7fa24c009040 comp rx=0 tx=0).stop 2024-03-28T15:10:35.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 -- 172.21.15.186:0/2770763188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa25c154bd0 msgr2=0x7fa25c153440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa25c154bd0 0x7fa25c153440 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa254009ee0 tx=0x7fa254004a40 comp rx=0 tx=0).stop 2024-03-28T15:10:35.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 -- 172.21.15.186:0/2770763188 shutdown_connections 2024-03-28T15:10:35.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fa23c06c530 0x7fa23c06e9f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa25c154bd0 0x7fa25c153440 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25c0c57e0 0x7fa25c15c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 --2- 172.21.15.186:0/2770763188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa25c0c53c0 0x7fa25c1f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 -- 172.21.15.186:0/2770763188 >> 172.21.15.186:0/2770763188 conn(0x7fa25c0731f0 msgr2=0x7fa25c0cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:35.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 -- 172.21.15.186:0/2770763188 shutdown_connections 2024-03-28T15:10:35.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.739+0000 7fa26115e640 1 -- 172.21.15.186:0/2770763188 wait complete. 2024-03-28T15:10:35.752 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:01:18.661274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:01:29.512445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:01:30.388340+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 33, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:10:35.753 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:10:35.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.856+0000 7f4e315e4640 1 Processor -- start 2024-03-28T15:10:35.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.856+0000 7f4e315e4640 1 -- start start 2024-03-28T15:10:35.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.856+0000 7f4e315e4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e2c0cc2b0 0x7f4e2c15de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.856+0000 7f4e315e4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e2c15a950 con 0x7f4e2c154af0 2024-03-28T15:10:35.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.856+0000 7f4e315e4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e2c15abc0 con 0x7f4e2c072780 2024-03-28T15:10:35.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.856+0000 7f4e315e4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4e2c15aea0 con 0x7f4e2c0cc2b0 2024-03-28T15:10:35.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.857+0000 7f4e2affd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4e2c154af0 0x7f4e2c0c5da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:33464/0 (socket says 172.21.15.186:33464) 2024-03-28T15:10:35.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.857+0000 7f4e2affd640 1 -- 172.21.15.186:0/1612576276 learned_addr learned my addr 172.21.15.186:0/1612576276 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:35.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.857+0000 7f4e2b7fe640 1 --2- 172.21.15.186:0/1612576276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e2c0cc2b0 0x7f4e2c15de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.857+0000 7f4e29ffb640 1 -- 172.21.15.186:0/1612576276 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3325821991 0 0) 0x7f4e2c15abc0 con 0x7f4e2c072780 2024-03-28T15:10:35.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.857+0000 7f4e2b7fe640 1 -- 172.21.15.186:0/1612576276 >> v1:172.21.15.191:6789/0 conn(0x7f4e2c072780 legacy=0x7f4e2c0d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.857+0000 7f4e2b7fe640 1 -- 172.21.15.186:0/1612576276 >> v1:172.21.15.186:6789/0 conn(0x7f4e2c154af0 legacy=0x7f4e2c0c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.857+0000 7f4e2b7fe640 1 -- 172.21.15.186:0/1612576276 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2c15b180 con 0x7f4e2c0cc2b0 2024-03-28T15:10:35.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.858+0000 7f4e2b7fe640 1 --2- 172.21.15.186:0/1612576276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e2c0cc2b0 0x7f4e2c15de30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009dc0 tx=0x7f4e2402bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13565329c382dda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:35.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.858+0000 7f4e29ffb640 1 -- 172.21.15.186:0/1612576276 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24039070 con 0x7f4e2c0cc2b0 2024-03-28T15:10:35.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.858+0000 7f4e29ffb640 1 -- 172.21.15.186:0/1612576276 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e2402cb10 con 0x7f4e2c0cc2b0 2024-03-28T15:10:35.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.858+0000 7f4e29ffb640 1 -- 172.21.15.186:0/1612576276 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24034440 con 0x7f4e2c0cc2b0 2024-03-28T15:10:35.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.858+0000 7f4e315e4640 1 -- 172.21.15.186:0/1612576276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e2c0cc2b0 msgr2=0x7f4e2c15de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.858+0000 7f4e315e4640 1 --2- 172.21.15.186:0/1612576276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e2c0cc2b0 0x7f4e2c15de30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009dc0 tx=0x7f4e2402bde0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.858+0000 7f4e315e4640 1 -- 172.21.15.186:0/1612576276 shutdown_connections 2024-03-28T15:10:35.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.858+0000 7f4e315e4640 1 --2- 172.21.15.186:0/1612576276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e2c0cc2b0 0x7f4e2c15de30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.858+0000 7f4e315e4640 1 -- 172.21.15.186:0/1612576276 >> 172.21.15.186:0/1612576276 conn(0x7f4e2c073580 msgr2=0x7f4e2c073990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:35.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.858+0000 7f4e315e4640 1 -- 172.21.15.186:0/1612576276 shutdown_connections 2024-03-28T15:10:35.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.858+0000 7f4e315e4640 1 -- 172.21.15.186:0/1612576276 wait complete. 2024-03-28T15:10:35.866 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e315e4640 1 Processor -- start 2024-03-28T15:10:35.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e315e4640 1 -- start start 2024-03-28T15:10:35.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e315e4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c072780 0x7f4e2c1f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e315e4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e2c0cc2b0 0x7f4e2c0cf220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e315e4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e2c154af0 0x7f4e2c0cf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e315e4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c0cd800 con 0x7f4e2c072780 2024-03-28T15:10:35.868 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e315e4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c154ed0 con 0x7f4e2c0cc2b0 2024-03-28T15:10:35.868 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e315e4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4e2c0d0d20 con 0x7f4e2c154af0 2024-03-28T15:10:35.868 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e2affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c072780 0x7f4e2c1f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.868 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e2affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c072780 0x7f4e2c1f3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52768/0 (socket says 172.21.15.186:52768) 2024-03-28T15:10:35.868 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e2affd640 1 -- 172.21.15.186:0/2222631849 learned_addr learned my addr 172.21.15.186:0/2222631849 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:35.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e2a7fc640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e2c0cc2b0 0x7f4e2c0cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.860+0000 7f4e2b7fe640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e2c154af0 0x7f4e2c0cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.861+0000 7f4e2affd640 1 -- 172.21.15.186:0/2222631849 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e2c154af0 msgr2=0x7f4e2c0cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.861+0000 7f4e2affd640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e2c154af0 0x7f4e2c0cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.861+0000 7f4e2affd640 1 -- 172.21.15.186:0/2222631849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e2c0cc2b0 msgr2=0x7f4e2c0cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:35.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.861+0000 7f4e2affd640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e2c0cc2b0 0x7f4e2c0cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:35.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.861+0000 7f4e2affd640 1 -- 172.21.15.186:0/2222631849 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2c1f4460 con 0x7f4e2c072780 2024-03-28T15:10:35.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.861+0000 7f4e2affd640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c072780 0x7f4e2c1f3b30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c00c5a0 tx=0x7f4e1c00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:35.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.861+0000 7f4e237fe640 1 -- 172.21.15.186:0/2222631849 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c017070 con 0x7f4e2c072780 2024-03-28T15:10:35.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.861+0000 7f4e237fe640 1 -- 172.21.15.186:0/2222631849 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e1c005ce0 con 0x7f4e2c072780 2024-03-28T15:10:35.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.861+0000 7f4e315e4640 1 -- 172.21.15.186:0/2222631849 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e2c153d80 con 0x7f4e2c072780 2024-03-28T15:10:35.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.861+0000 7f4e237fe640 1 -- 172.21.15.186:0/2222631849 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c0072b0 con 0x7f4e2c072780 2024-03-28T15:10:35.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.861+0000 7f4e315e4640 1 -- 172.21.15.186:0/2222631849 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e2c158720 con 0x7f4e2c072780 2024-03-28T15:10:35.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.862+0000 7f4e315e4640 1 -- 172.21.15.186:0/2222631849 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e2c158920 con 0x7f4e2c072780 2024-03-28T15:10:35.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.862+0000 7f4e237fe640 1 -- 172.21.15.186:0/2222631849 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4e1c01a070 con 0x7f4e2c072780 2024-03-28T15:10:35.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.863+0000 7f4e237fe640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c560 0x7f4e0c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:35.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.863+0000 7f4e237fe640 1 -- 172.21.15.186:0/2222631849 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f4e1c08aca0 con 0x7f4e2c072780 2024-03-28T15:10:35.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.863+0000 7f4e2a7fc640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c560 0x7f4e0c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:35.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.863+0000 7f4e2a7fc640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c560 0x7f4e0c06ea20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c1f4980 tx=0x7f4e14009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:35.873 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:35.867+0000 7f4e237fe640 1 -- 172.21.15.186:0/2222631849 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4e1c058a50 con 0x7f4e2c072780 2024-03-28T15:10:36.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.042+0000 7f4e315e4640 1 -- 172.21.15.186:0/2222631849 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4e2c069570 con 0x7f4e2c072780 2024-03-28T15:10:36.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.042+0000 7f4e237fe640 1 -- 172.21.15.186:0/2222631849 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f4e1c05cd50 con 0x7f4e2c072780 2024-03-28T15:10:36.043 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 1 2024-03-28T15:10:36.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 -- 172.21.15.186:0/2222631849 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c560 msgr2=0x7f4e0c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c560 0x7f4e0c06ea20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c1f4980 tx=0x7f4e14009040 comp rx=0 tx=0).stop 2024-03-28T15:10:36.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 -- 172.21.15.186:0/2222631849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c072780 msgr2=0x7f4e2c1f3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c072780 0x7f4e2c1f3b30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c00c5a0 tx=0x7f4e1c00ca60 comp rx=0 tx=0).stop 2024-03-28T15:10:36.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 -- 172.21.15.186:0/2222631849 shutdown_connections 2024-03-28T15:10:36.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c560 0x7f4e0c06ea20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e2c154af0 0x7f4e2c0cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e2c0cc2b0 0x7f4e2c0cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 --2- 172.21.15.186:0/2222631849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c072780 0x7f4e2c1f3b30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 -- 172.21.15.186:0/2222631849 >> 172.21.15.186:0/2222631849 conn(0x7f4e2c073580 msgr2=0x7f4e2c149840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:36.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 -- 172.21.15.186:0/2222631849 shutdown_connections 2024-03-28T15:10:36.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.044+0000 7f4e315e4640 1 -- 172.21.15.186:0/2222631849 wait complete. 2024-03-28T15:10:36.056 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:10:36.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.161+0000 7f0ef0083640 1 Processor -- start 2024-03-28T15:10:36.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.162+0000 7f0ef0083640 1 -- start start 2024-03-28T15:10:36.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.162+0000 7f0ef0083640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0ee8154a10 0x7f0ee80cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.162+0000 7f0ef0083640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ee80cce90 0x7f0ee80cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.162+0000 7f0ef0083640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee815de90 con 0x7f0ee815b2c0 2024-03-28T15:10:36.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.162+0000 7f0ef0083640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee80c5ae0 con 0x7f0ee8154a10 2024-03-28T15:10:36.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.162+0000 7f0ef0083640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee815e750 con 0x7f0ee80cce90 2024-03-28T15:10:36.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.163+0000 7f0eede1f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0ee8154a10 0x7f0ee80cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.163+0000 7f0eed61e640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f0ee815b2c0 0x7f0ee80c53d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:33480/0 (socket says 172.21.15.186:33480) 2024-03-28T15:10:36.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.163+0000 7f0eed61e640 1 -- 172.21.15.186:0/3495733942 learned_addr learned my addr 172.21.15.186:0/3495733942 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:36.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.163+0000 7f0eede1f640 1 --2- 172.21.15.186:0/3495733942 >> v2:172.21.15.186:3300/0 conn(0x7f0ee8154a10 0x7f0ee80cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52784/0 (socket says 172.21.15.186:52784) 2024-03-28T15:10:36.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.163+0000 7f0eee620640 1 --2- 172.21.15.186:0/3495733942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ee80cce90 0x7f0ee80cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.163+0000 7f0eede1f640 1 -- 172.21.15.186:0/3495733942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ee80cce90 msgr2=0x7f0ee80cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:36.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.163+0000 7f0eede1f640 1 --2- 172.21.15.186:0/3495733942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ee80cce90 0x7f0ee80cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.163+0000 7f0eede1f640 1 -- 172.21.15.186:0/3495733942 >> v1:172.21.15.186:6789/0 conn(0x7f0ee815b2c0 legacy=0x7f0ee80c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.163+0000 7f0eede1f640 1 -- 172.21.15.186:0/3495733942 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee80cb6d0 con 0x7f0ee8154a10 2024-03-28T15:10:36.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.163+0000 7f0eece1d640 1 -- 172.21.15.186:0/3495733942 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3116446201 0 0) 0x7f0ee815de90 con 0x7f0ee815b2c0 2024-03-28T15:10:36.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0eede1f640 1 --2- 172.21.15.186:0/3495733942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee8154a10 0x7f0ee80cb0b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40097a0 tx=0x7f0ee402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bc1c712cf06db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:36.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0eece1d640 1 -- 172.21.15.186:0/3495733942 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4034070 con 0x7f0ee8154a10 2024-03-28T15:10:36.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0eece1d640 1 -- 172.21.15.186:0/3495733942 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ee4030610 con 0x7f0ee8154a10 2024-03-28T15:10:36.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0eece1d640 1 -- 172.21.15.186:0/3495733942 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee40309e0 con 0x7f0ee8154a10 2024-03-28T15:10:36.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0ef0083640 1 -- 172.21.15.186:0/3495733942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee8154a10 msgr2=0x7f0ee80cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0ef0083640 1 --2- 172.21.15.186:0/3495733942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee8154a10 0x7f0ee80cb0b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40097a0 tx=0x7f0ee402b960 comp rx=0 tx=0).stop 2024-03-28T15:10:36.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0ef0083640 1 -- 172.21.15.186:0/3495733942 shutdown_connections 2024-03-28T15:10:36.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0ef0083640 1 --2- 172.21.15.186:0/3495733942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ee80cce90 0x7f0ee80cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0ef0083640 1 --2- 172.21.15.186:0/3495733942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee8154a10 0x7f0ee80cb0b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0ef0083640 1 -- 172.21.15.186:0/3495733942 >> 172.21.15.186:0/3495733942 conn(0x7f0ee806b640 msgr2=0x7f0ee806ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:36.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0ef0083640 1 -- 172.21.15.186:0/3495733942 shutdown_connections 2024-03-28T15:10:36.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.164+0000 7f0ef0083640 1 -- 172.21.15.186:0/3495733942 wait complete. 2024-03-28T15:10:36.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0ef0083640 1 Processor -- start 2024-03-28T15:10:36.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0ef0083640 1 -- start start 2024-03-28T15:10:36.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0ef0083640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ee80cce90 0x7f0ee8150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0ef0083640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ee8154a10 0x7f0ee80cf690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0ef0083640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee815b2c0 0x7f0ee814a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0ef0083640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee80cb9b0 con 0x7f0ee815b2c0 2024-03-28T15:10:36.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0ef0083640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee815e3e0 con 0x7f0ee80cce90 2024-03-28T15:10:36.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0ef0083640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee815b710 con 0x7f0ee8154a10 2024-03-28T15:10:36.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0eee620640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee815b2c0 0x7f0ee814a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0eee620640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee815b2c0 0x7f0ee814a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52796/0 (socket says 172.21.15.186:52796) 2024-03-28T15:10:36.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0eee620640 1 -- 172.21.15.186:0/3025784021 learned_addr learned my addr 172.21.15.186:0/3025784021 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:36.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0eed61e640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ee8154a10 0x7f0ee80cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.166+0000 7f0eede1f640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ee80cce90 0x7f0ee8150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.167+0000 7f0eee620640 1 -- 172.21.15.186:0/3025784021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ee8154a10 msgr2=0x7f0ee80cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.167+0000 7f0eee620640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ee8154a10 0x7f0ee80cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.167+0000 7f0eee620640 1 -- 172.21.15.186:0/3025784021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ee80cce90 msgr2=0x7f0ee8150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.167+0000 7f0eee620640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ee80cce90 0x7f0ee8150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.167+0000 7f0eee620640 1 -- 172.21.15.186:0/3025784021 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee814ec90 con 0x7f0ee815b2c0 2024-03-28T15:10:36.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.167+0000 7f0eee620640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee815b2c0 0x7f0ee814a4d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000efc0 tx=0x7f0ee000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:36.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.167+0000 7f0ede7fc640 1 -- 172.21.15.186:0/3025784021 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0018020 con 0x7f0ee815b2c0 2024-03-28T15:10:36.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.167+0000 7f0ede7fc640 1 -- 172.21.15.186:0/3025784021 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ee0014480 con 0x7f0ee815b2c0 2024-03-28T15:10:36.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.167+0000 7f0ef0083640 1 -- 172.21.15.186:0/3025784021 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee80c7ae0 con 0x7f0ee815b2c0 2024-03-28T15:10:36.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.167+0000 7f0ede7fc640 1 -- 172.21.15.186:0/3025784021 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0014870 con 0x7f0ee815b2c0 2024-03-28T15:10:36.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.167+0000 7f0ef0083640 1 -- 172.21.15.186:0/3025784021 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee80ce360 con 0x7f0ee815b2c0 2024-03-28T15:10:36.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.168+0000 7f0ef0083640 1 -- 172.21.15.186:0/3025784021 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee8069570 con 0x7f0ee815b2c0 2024-03-28T15:10:36.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.168+0000 7f0ede7fc640 1 -- 172.21.15.186:0/3025784021 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0ee001b020 con 0x7f0ee815b2c0 2024-03-28T15:10:36.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.169+0000 7f0ede7fc640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ecc06c530 0x7f0ecc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.169+0000 7f0ede7fc640 1 -- 172.21.15.186:0/3025784021 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f0ee008c280 con 0x7f0ee815b2c0 2024-03-28T15:10:36.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.169+0000 7f0eede1f640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ecc06c530 0x7f0ecc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.169+0000 7f0eede1f640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ecc06c530 0x7f0ecc06e9f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4036000 tx=0x7f0ee403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:36.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.173+0000 7f0ede7fc640 1 -- 172.21.15.186:0/3025784021 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f0ee0022030 con 0x7f0ee815b2c0 2024-03-28T15:10:36.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.347+0000 7f0ef0083640 1 -- 172.21.15.186:0/3025784021 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0ee80cbc60 con 0x7f0ee815b2c0 2024-03-28T15:10:36.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.347+0000 7f0ede7fc640 1 -- 172.21.15.186:0/3025784021 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) ==== 104+0+11 (secure 0 0 0) 0x7f0ee0010040 con 0x7f0ee815b2c0 2024-03-28T15:10:36.348 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 32 2024-03-28T15:10:36.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.349+0000 7f0ef0083640 1 -- 172.21.15.186:0/3025784021 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ecc06c530 msgr2=0x7f0ecc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.349+0000 7f0ef0083640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ecc06c530 0x7f0ecc06e9f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4036000 tx=0x7f0ee403b040 comp rx=0 tx=0).stop 2024-03-28T15:10:36.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.349+0000 7f0ef0083640 1 -- 172.21.15.186:0/3025784021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee815b2c0 msgr2=0x7f0ee814a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.349+0000 7f0ef0083640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee815b2c0 0x7f0ee814a4d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000efc0 tx=0x7f0ee000c490 comp rx=0 tx=0).stop 2024-03-28T15:10:36.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.349+0000 7f0ef0083640 1 -- 172.21.15.186:0/3025784021 shutdown_connections 2024-03-28T15:10:36.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.349+0000 7f0ef0083640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ecc06c530 0x7f0ecc06e9f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.349+0000 7f0ef0083640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee815b2c0 0x7f0ee814a4d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.349+0000 7f0ef0083640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ee8154a10 0x7f0ee80cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.349+0000 7f0ef0083640 1 --2- 172.21.15.186:0/3025784021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ee80cce90 0x7f0ee8150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.349+0000 7f0ef0083640 1 -- 172.21.15.186:0/3025784021 >> 172.21.15.186:0/3025784021 conn(0x7f0ee806b640 msgr2=0x7f0ee8153c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:36.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.350+0000 7f0ef0083640 1 -- 172.21.15.186:0/3025784021 shutdown_connections 2024-03-28T15:10:36.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.350+0000 7f0ef0083640 1 -- 172.21.15.186:0/3025784021 wait complete. 2024-03-28T15:10:36.362 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:10:36.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.464+0000 7f97ecb93640 1 Processor -- start 2024-03-28T15:10:36.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.464+0000 7f97ecb93640 1 -- start start 2024-03-28T15:10:36.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.465+0000 7f97ecb93640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97e8154a10 0x7f97e80cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.465+0000 7f97ecb93640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f97e815b2c0 0x7f97e80c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.465+0000 7f97ecb93640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97e815de90 con 0x7f97e80cce90 2024-03-28T15:10:36.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.465+0000 7f97ecb93640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f97e815e750 con 0x7f97e815b2c0 2024-03-28T15:10:36.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.465+0000 7f97ecb93640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f97e80cb6d0 con 0x7f97e8154a10 2024-03-28T15:10:36.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.465+0000 7f97e6d9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f97e80cce90 0x7f97e8071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:33486/0 (socket says 172.21.15.186:33486) 2024-03-28T15:10:36.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.465+0000 7f97e6d9d640 1 -- 172.21.15.186:0/2878677043 learned_addr learned my addr 172.21.15.186:0/2878677043 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:36.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.465+0000 7f97e5d9b640 1 --2- 172.21.15.186:0/2878677043 >> v2:172.21.15.191:3300/0 conn(0x7f97e815b2c0 0x7f97e80c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.465+0000 7f97e659c640 1 --2- 172.21.15.186:0/2878677043 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97e8154a10 0x7f97e80cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.466+0000 7f97e559a640 1 -- 172.21.15.186:0/2878677043 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2557844843 0 0) 0x7f97e815de90 con 0x7f97e80cce90 2024-03-28T15:10:36.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.466+0000 7f97e559a640 1 -- 172.21.15.186:0/2878677043 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97d0003520 con 0x7f97e80cce90 2024-03-28T15:10:36.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.466+0000 7f97e5d9b640 1 -- 172.21.15.186:0/2878677043 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97e8154a10 msgr2=0x7f97e80cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:36.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.466+0000 7f97e5d9b640 1 --2- 172.21.15.186:0/2878677043 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97e8154a10 0x7f97e80cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.466+0000 7f97e5d9b640 1 -- 172.21.15.186:0/2878677043 >> v1:172.21.15.186:6789/0 conn(0x7f97e80cce90 legacy=0x7f97e8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.466+0000 7f97e5d9b640 1 -- 172.21.15.186:0/2878677043 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e815e100 con 0x7f97e815b2c0 2024-03-28T15:10:36.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.466+0000 7f97e659c640 1 --2- 172.21.15.186:0/2878677043 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97e8154a10 0x7f97e80cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:36.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97e5d9b640 1 --2- 172.21.15.186:0/2878677043 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97e815b2c0 0x7f97e80c53d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0097a0 tx=0x7f97dc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56d65bdccafa048e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:36.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97e559a640 1 -- 172.21.15.186:0/2878677043 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc034070 con 0x7f97e815b2c0 2024-03-28T15:10:36.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97e559a640 1 -- 172.21.15.186:0/2878677043 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97dc030610 con 0x7f97e815b2c0 2024-03-28T15:10:36.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97e559a640 1 -- 172.21.15.186:0/2878677043 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc0309c0 con 0x7f97e815b2c0 2024-03-28T15:10:36.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97ecb93640 1 -- 172.21.15.186:0/2878677043 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97e815b2c0 msgr2=0x7f97e80c53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97ecb93640 1 --2- 172.21.15.186:0/2878677043 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97e815b2c0 0x7f97e80c53d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0097a0 tx=0x7f97dc02b960 comp rx=0 tx=0).stop 2024-03-28T15:10:36.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97ecb93640 1 -- 172.21.15.186:0/2878677043 shutdown_connections 2024-03-28T15:10:36.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97ecb93640 1 --2- 172.21.15.186:0/2878677043 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97e815b2c0 0x7f97e80c53d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97ecb93640 1 --2- 172.21.15.186:0/2878677043 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97e8154a10 0x7f97e80cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97ecb93640 1 -- 172.21.15.186:0/2878677043 >> 172.21.15.186:0/2878677043 conn(0x7f97e806b640 msgr2=0x7f97e806ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:36.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97ecb93640 1 -- 172.21.15.186:0/2878677043 shutdown_connections 2024-03-28T15:10:36.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.467+0000 7f97ecb93640 1 -- 172.21.15.186:0/2878677043 wait complete. 2024-03-28T15:10:36.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97ecb93640 1 Processor -- start 2024-03-28T15:10:36.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97ecb93640 1 -- start start 2024-03-28T15:10:36.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97ecb93640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97e80cce90 0x7f97e80c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97ecb93640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97e8154a10 0x7f97e81e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97ecb93640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e815b2c0 0x7f97e815cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97ecb93640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97e815b6e0 con 0x7f97e815b2c0 2024-03-28T15:10:36.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97ecb93640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f97e80cb9b0 con 0x7f97e80cce90 2024-03-28T15:10:36.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97ecb93640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f97e815ebe0 con 0x7f97e8154a10 2024-03-28T15:10:36.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97e659c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97e80cce90 0x7f97e80c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97e6d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e815b2c0 0x7f97e815cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97e5d9b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97e8154a10 0x7f97e81e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97e659c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97e80cce90 0x7f97e80c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:43866/0 (socket says 172.21.15.186:43866) 2024-03-28T15:10:36.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.469+0000 7f97e659c640 1 -- 172.21.15.186:0/2929327602 learned_addr learned my addr 172.21.15.186:0/2929327602 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:36.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97e659c640 1 -- 172.21.15.186:0/2929327602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97e8154a10 msgr2=0x7f97e81e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97e659c640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97e8154a10 0x7f97e81e4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97e659c640 1 -- 172.21.15.186:0/2929327602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e815b2c0 msgr2=0x7f97e815cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97e659c640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e815b2c0 0x7f97e815cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97e659c640 1 -- 172.21.15.186:0/2929327602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e81e3de0 con 0x7f97e80cce90 2024-03-28T15:10:36.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97e6d9d640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e815b2c0 0x7f97e815cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:36.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97e659c640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97e80cce90 0x7f97e80c7400 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f97d800efc0 tx=0x7f97d800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:36.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97d6ffd640 1 -- 172.21.15.186:0/2929327602 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8018020 con 0x7f97e80cce90 2024-03-28T15:10:36.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97d6ffd640 1 -- 172.21.15.186:0/2929327602 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97d8014680 con 0x7f97e80cce90 2024-03-28T15:10:36.475 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97ecb93640 1 -- 172.21.15.186:0/2929327602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e81e4080 con 0x7f97e80cce90 2024-03-28T15:10:36.476 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97d6ffd640 1 -- 172.21.15.186:0/2929327602 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8014a50 con 0x7f97e80cce90 2024-03-28T15:10:36.476 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.470+0000 7f97ecb93640 1 -- 172.21.15.186:0/2929327602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e81f0530 con 0x7f97e80cce90 2024-03-28T15:10:36.476 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.471+0000 7f97ecb93640 1 -- 172.21.15.186:0/2929327602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97e81f0040 con 0x7f97e80cce90 2024-03-28T15:10:36.476 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.471+0000 7f97d6ffd640 1 -- 172.21.15.186:0/2929327602 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f97d8005d10 con 0x7f97e80cce90 2024-03-28T15:10:36.476 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.472+0000 7f97d6ffd640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97c806c560 0x7f97c806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.476 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.472+0000 7f97d6ffd640 1 -- 172.21.15.186:0/2929327602 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f97d808c6f0 con 0x7f97e80cce90 2024-03-28T15:10:36.476 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.472+0000 7f97e5d9b640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97c806c560 0x7f97c806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.477 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.472+0000 7f97e5d9b640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97c806c560 0x7f97c806ea20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97dc036000 tx=0x7f97dc03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:36.477 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.477+0000 7f97d6ffd640 1 -- 172.21.15.186:0/2929327602 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f97d8022030 con 0x7f97e80cce90 2024-03-28T15:10:36.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.652+0000 7f97ecb93640 1 -- 172.21.15.186:0/2929327602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f97e8069570 con 0x7f97e80cce90 2024-03-28T15:10:36.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.652+0000 7f97d6ffd640 1 -- 172.21.15.186:0/2929327602 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) ==== 100+0+11 (secure 0 0 0) 0x7f97d808ae20 con 0x7f97e80cce90 2024-03-28T15:10:36.653 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 64 2024-03-28T15:10:36.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 -- 172.21.15.186:0/2929327602 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97c806c560 msgr2=0x7f97c806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97c806c560 0x7f97c806ea20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97dc036000 tx=0x7f97dc03b040 comp rx=0 tx=0).stop 2024-03-28T15:10:36.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 -- 172.21.15.186:0/2929327602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97e80cce90 msgr2=0x7f97e80c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97e80cce90 0x7f97e80c7400 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f97d800efc0 tx=0x7f97d800c490 comp rx=0 tx=0).stop 2024-03-28T15:10:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 -- 172.21.15.186:0/2929327602 shutdown_connections 2024-03-28T15:10:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f97c806c560 0x7f97c806ea20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97e815b2c0 0x7f97e815cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97e8154a10 0x7f97e81e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 --2- 172.21.15.186:0/2929327602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97e80cce90 0x7f97e80c7400 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 -- 172.21.15.186:0/2929327602 >> 172.21.15.186:0/2929327602 conn(0x7f97e806b640 msgr2=0x7f97e81521e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:36.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 -- 172.21.15.186:0/2929327602 shutdown_connections 2024-03-28T15:10:36.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.654+0000 7f97ecb93640 1 -- 172.21.15.186:0/2929327602 wait complete. 2024-03-28T15:10:36.667 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T15:10:36.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.769+0000 7f327e217640 1 Processor -- start 2024-03-28T15:10:36.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.769+0000 7f327e217640 1 -- start start 2024-03-28T15:10:36.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f327e217640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f32780c2800 0x7f32780c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f327e217640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f327806c940 0x7f32780c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f327e217640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32780c7490 con 0x7f3278150fd0 2024-03-28T15:10:36.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f327e217640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f32780c1760 con 0x7f327806c940 2024-03-28T15:10:36.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f327e217640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f32780c1a40 con 0x7f32780c2800 2024-03-28T15:10:36.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f3277fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f327806c940 0x7f32780c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f3277fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f327806c940 0x7f32780c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52822/0 (socket says 172.21.15.186:52822) 2024-03-28T15:10:36.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f3277fff640 1 -- 172.21.15.186:0/3347446410 learned_addr learned my addr 172.21.15.186:0/3347446410 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:36.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f3276ffd640 1 --2- 172.21.15.186:0/3347446410 >> v2:172.21.15.191:3300/0 conn(0x7f32780c2800 0x7f32780c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f3277fff640 1 -- 172.21.15.186:0/3347446410 >> v2:172.21.15.191:3300/0 conn(0x7f32780c2800 msgr2=0x7f32780c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:36.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f3277fff640 1 --2- 172.21.15.186:0/3347446410 >> v2:172.21.15.191:3300/0 conn(0x7f32780c2800 0x7f32780c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f3277fff640 1 -- 172.21.15.186:0/3347446410 >> v1:172.21.15.191:6789/0 conn(0x7f3278150fd0 legacy=0x7f327815ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.770+0000 7f3277fff640 1 -- 172.21.15.186:0/3347446410 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32780c1d20 con 0x7f327806c940 2024-03-28T15:10:36.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.771+0000 7f3277fff640 1 --2- 172.21.15.186:0/3347446410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f327806c940 0x7f32780c7b40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3278071990 tx=0x7f327002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37cf0fbd17268bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:36.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.771+0000 7f32767fc640 1 -- 172.21.15.186:0/3347446410 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327002cb20 con 0x7f327806c940 2024-03-28T15:10:36.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.771+0000 7f32767fc640 1 -- 172.21.15.186:0/3347446410 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f327002ccc0 con 0x7f327806c940 2024-03-28T15:10:36.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.771+0000 7f32767fc640 1 -- 172.21.15.186:0/3347446410 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270034610 con 0x7f327806c940 2024-03-28T15:10:36.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.771+0000 7f327e217640 1 -- 172.21.15.186:0/3347446410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f327806c940 msgr2=0x7f32780c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.771+0000 7f327e217640 1 --2- 172.21.15.186:0/3347446410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f327806c940 0x7f32780c7b40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3278071990 tx=0x7f327002bf80 comp rx=0 tx=0).stop 2024-03-28T15:10:36.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.772+0000 7f327e217640 1 -- 172.21.15.186:0/3347446410 shutdown_connections 2024-03-28T15:10:36.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.772+0000 7f327e217640 1 --2- 172.21.15.186:0/3347446410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f327806c940 0x7f32780c7b40 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.772+0000 7f327e217640 1 --2- 172.21.15.186:0/3347446410 >> v2:172.21.15.191:3300/0 conn(0x7f32780c2800 0x7f32780c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.772+0000 7f327e217640 1 -- 172.21.15.186:0/3347446410 >> 172.21.15.186:0/3347446410 conn(0x7f327806c010 msgr2=0x7f32780cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:36.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.772+0000 7f327e217640 1 -- 172.21.15.186:0/3347446410 shutdown_connections 2024-03-28T15:10:36.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.772+0000 7f327e217640 1 -- 172.21.15.186:0/3347446410 wait complete. 2024-03-28T15:10:36.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.773+0000 7f327e217640 1 Processor -- start 2024-03-28T15:10:36.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.773+0000 7f327e217640 1 -- start start 2024-03-28T15:10:36.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.773+0000 7f327e217640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f327806c940 0x7f327813bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.773+0000 7f327e217640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32780c2800 0x7f32781456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f327e217640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3278150fd0 0x7f32781625a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f327e217640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f32781513b0 con 0x7f3278150fd0 2024-03-28T15:10:36.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f327e217640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f32780c22e0 con 0x7f32780c2800 2024-03-28T15:10:36.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f327e217640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f32780c8790 con 0x7f327806c940 2024-03-28T15:10:36.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f3277fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3278150fd0 0x7f32781625a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f3277fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3278150fd0 0x7f32781625a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52826/0 (socket says 172.21.15.186:52826) 2024-03-28T15:10:36.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f3277fff640 1 -- 172.21.15.186:0/148116550 learned_addr learned my addr 172.21.15.186:0/148116550 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:36.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f32777fe640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f327806c940 0x7f327813bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f3276ffd640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32780c2800 0x7f32781456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f3277fff640 1 -- 172.21.15.186:0/148116550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f327806c940 msgr2=0x7f327813bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f3277fff640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f327806c940 0x7f327813bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f3277fff640 1 -- 172.21.15.186:0/148116550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32780c2800 msgr2=0x7f32781456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f3277fff640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32780c2800 0x7f32781456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f3277fff640 1 -- 172.21.15.186:0/148116550 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3278073440 con 0x7f3278150fd0 2024-03-28T15:10:36.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f3277fff640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3278150fd0 0x7f32781625a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3270009f40 tx=0x7f3270004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:36.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f325ffff640 1 -- 172.21.15.186:0/148116550 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270043020 con 0x7f3278150fd0 2024-03-28T15:10:36.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f325ffff640 1 -- 172.21.15.186:0/148116550 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f327003e450 con 0x7f3278150fd0 2024-03-28T15:10:36.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f327e217640 1 -- 172.21.15.186:0/148116550 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f327813cc40 con 0x7f3278150fd0 2024-03-28T15:10:36.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f325ffff640 1 -- 172.21.15.186:0/148116550 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270009510 con 0x7f3278150fd0 2024-03-28T15:10:36.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.774+0000 7f327e217640 1 -- 172.21.15.186:0/148116550 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3278072680 con 0x7f3278150fd0 2024-03-28T15:10:36.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.775+0000 7f327e217640 1 -- 172.21.15.186:0/148116550 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3278069570 con 0x7f3278150fd0 2024-03-28T15:10:36.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.776+0000 7f325ffff640 1 -- 172.21.15.186:0/148116550 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3270046020 con 0x7f3278150fd0 2024-03-28T15:10:36.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.776+0000 7f325ffff640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f326406c530 0x7f326406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:36.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.776+0000 7f325ffff640 1 -- 172.21.15.186:0/148116550 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f32700ae330 con 0x7f3278150fd0 2024-03-28T15:10:36.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.776+0000 7f32777fe640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f326406c530 0x7f326406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:36.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.777+0000 7f32777fe640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f326406c530 0x7f326406e9f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f32680016e0 tx=0x7f3268009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:36.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.781+0000 7f325ffff640 1 -- 172.21.15.186:0/148116550 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f327004d020 con 0x7f3278150fd0 2024-03-28T15:10:36.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.969+0000 7f327e217640 1 -- 172.21.15.186:0/148116550 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3278071520 con 0x7f3278150fd0 2024-03-28T15:10:36.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.969+0000 7f325ffff640 1 -- 172.21.15.186:0/148116550 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v37) ==== 81+0+618 (secure 0 0 0) 0x7f3278073440 con 0x7f3278150fd0 2024-03-28T15:10:36.970 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6834/1992706523 2024-03-29T15:08:31.220829+0000 2024-03-28T15:10:36.970 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6837/3037586083 2024-03-29T15:09:56.237602+0000 2024-03-28T15:10:36.971 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6835/3037586083 2024-03-29T15:09:56.237602+0000 2024-03-28T15:10:36.971 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6837/3828930138 2024-03-29T15:06:06.202259+0000 2024-03-28T15:10:36.971 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6835/3828930138 2024-03-29T15:06:06.202259+0000 2024-03-28T15:10:36.971 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6832/1992706523 2024-03-29T15:08:31.220829+0000 2024-03-28T15:10:36.971 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6834/185536353 2024-03-29T15:03:46.182756+0000 2024-03-28T15:10:36.971 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6832/185536353 2024-03-29T15:03:46.182756+0000 2024-03-28T15:10:36.971 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6837/2813406930 2024-03-29T15:02:11.169402+0000 2024-03-28T15:10:36.971 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6835/2813406930 2024-03-29T15:02:11.169402+0000 2024-03-28T15:10:36.972 INFO:teuthology.orchestra.run.smithi186.stderr:listed 10 entries 2024-03-28T15:10:36.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 -- 172.21.15.186:0/148116550 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f326406c530 msgr2=0x7f326406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f326406c530 0x7f326406e9f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f32680016e0 tx=0x7f3268009040 comp rx=0 tx=0).stop 2024-03-28T15:10:36.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 -- 172.21.15.186:0/148116550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3278150fd0 msgr2=0x7f32781625a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:36.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3278150fd0 0x7f32781625a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3270009f40 tx=0x7f3270004a40 comp rx=0 tx=0).stop 2024-03-28T15:10:36.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 -- 172.21.15.186:0/148116550 shutdown_connections 2024-03-28T15:10:36.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f326406c530 0x7f326406e9f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3278150fd0 0x7f32781625a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32780c2800 0x7f32781456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 --2- 172.21.15.186:0/148116550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f327806c940 0x7f327813bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:36.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 -- 172.21.15.186:0/148116550 >> 172.21.15.186:0/148116550 conn(0x7f327806c010 msgr2=0x7f327814bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:36.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 -- 172.21.15.186:0/148116550 shutdown_connections 2024-03-28T15:10:36.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:36.971+0000 7f327e217640 1 -- 172.21.15.186:0/148116550 wait complete. 2024-03-28T15:10:36.983 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-28T15:10:36.983 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T15:10:36.998 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:10:37.138 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.137+0000 7f482cc81640 1 Processor -- start 2024-03-28T15:10:37.138 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.137+0000 7f482cc81640 1 -- start start 2024-03-28T15:10:37.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.138+0000 7f482cc81640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4828151990 0x7f48280c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.138+0000 7f482cc81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48280c2d00 0x7f48280c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.138+0000 7f482cc81640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f482806caa0 con 0x7f48280b1e30 2024-03-28T15:10:37.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.138+0000 7f482cc81640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f48280c1da0 con 0x7f4828151990 2024-03-28T15:10:37.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.138+0000 7f482cc81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f482806cd80 con 0x7f48280c2d00 2024-03-28T15:10:37.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.139+0000 7f482659c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4828151990 0x7f48280c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.139+0000 7f4825d9b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f48280b1e30 0x7f48280712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:33490/0 (socket says 172.21.15.186:33490) 2024-03-28T15:10:37.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.139+0000 7f4825d9b640 1 -- 172.21.15.186:0/3452068100 learned_addr learned my addr 172.21.15.186:0/3452068100 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:37.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.139+0000 7f4826d9d640 1 --2- 172.21.15.186:0/3452068100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48280c2d00 0x7f48280c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.139+0000 7f482659c640 1 -- 172.21.15.186:0/3452068100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48280c2d00 msgr2=0x7f48280c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:37.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.139+0000 7f482659c640 1 --2- 172.21.15.186:0/3452068100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48280c2d00 0x7f48280c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.139+0000 7f482659c640 1 -- 172.21.15.186:0/3452068100 >> v1:172.21.15.186:6789/0 conn(0x7f48280b1e30 legacy=0x7f48280712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.139+0000 7f482659c640 1 -- 172.21.15.186:0/3452068100 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48280c1340 con 0x7f4828151990 2024-03-28T15:10:37.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.139+0000 7f482559a640 1 -- 172.21.15.186:0/3452068100 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4262386112 0 0) 0x7f482806caa0 con 0x7f48280b1e30 2024-03-28T15:10:37.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482659c640 1 --2- 172.21.15.186:0/3452068100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4828151990 0x7f48280c2230 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f48200097a0 tx=0x7f482002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9706758bf97139 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:37.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482559a640 1 -- 172.21.15.186:0/3452068100 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820034070 con 0x7f4828151990 2024-03-28T15:10:37.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482559a640 1 -- 172.21.15.186:0/3452068100 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4820030610 con 0x7f4828151990 2024-03-28T15:10:37.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482559a640 1 -- 172.21.15.186:0/3452068100 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48200309e0 con 0x7f4828151990 2024-03-28T15:10:37.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482cc81640 1 -- 172.21.15.186:0/3452068100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4828151990 msgr2=0x7f48280c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482cc81640 1 --2- 172.21.15.186:0/3452068100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4828151990 0x7f48280c2230 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f48200097a0 tx=0x7f482002b960 comp rx=0 tx=0).stop 2024-03-28T15:10:37.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482cc81640 1 -- 172.21.15.186:0/3452068100 shutdown_connections 2024-03-28T15:10:37.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482cc81640 1 --2- 172.21.15.186:0/3452068100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48280c2d00 0x7f48280c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482cc81640 1 --2- 172.21.15.186:0/3452068100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4828151990 0x7f48280c2230 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482cc81640 1 -- 172.21.15.186:0/3452068100 >> 172.21.15.186:0/3452068100 conn(0x7f48280c8a20 msgr2=0x7f48280c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:37.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482cc81640 1 -- 172.21.15.186:0/3452068100 shutdown_connections 2024-03-28T15:10:37.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.140+0000 7f482cc81640 1 -- 172.21.15.186:0/3452068100 wait complete. 2024-03-28T15:10:37.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.142+0000 7f482cc81640 1 Processor -- start 2024-03-28T15:10:37.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.142+0000 7f482cc81640 1 -- start start 2024-03-28T15:10:37.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.142+0000 7f482cc81640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48280b1e30 0x7f482814dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.142+0000 7f482cc81640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48280c2d00 0x7f482814cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.142+0000 7f482cc81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4828151990 0x7f482814d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.142+0000 7f482cc81640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4828073dd0 con 0x7f48280c2d00 2024-03-28T15:10:37.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.142+0000 7f482cc81640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f48280c1900 con 0x7f48280b1e30 2024-03-28T15:10:37.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.142+0000 7f482cc81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4828073f50 con 0x7f4828151990 2024-03-28T15:10:37.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4825d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48280c2d00 0x7f482814cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4825d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48280c2d00 0x7f482814cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52856/0 (socket says 172.21.15.186:52856) 2024-03-28T15:10:37.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4825d9b640 1 -- 172.21.15.186:0/3396155204 learned_addr learned my addr 172.21.15.186:0/3396155204 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:37.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4826d9d640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4828151990 0x7f482814d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f482659c640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48280b1e30 0x7f482814dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4825d9b640 1 -- 172.21.15.186:0/3396155204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4828151990 msgr2=0x7f482814d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4825d9b640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4828151990 0x7f482814d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4825d9b640 1 -- 172.21.15.186:0/3396155204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48280b1e30 msgr2=0x7f482814dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4825d9b640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48280b1e30 0x7f482814dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4825d9b640 1 -- 172.21.15.186:0/3396155204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482814e130 con 0x7f48280c2d00 2024-03-28T15:10:37.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4825d9b640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48280c2d00 0x7f482814cc80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f481c00c7b0 tx=0x7f481c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:37.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4816ffd640 1 -- 172.21.15.186:0/3396155204 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c018070 con 0x7f48280c2d00 2024-03-28T15:10:37.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4816ffd640 1 -- 172.21.15.186:0/3396155204 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f481c00a640 con 0x7f48280c2d00 2024-03-28T15:10:37.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f482cc81640 1 -- 172.21.15.186:0/3396155204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48281587a0 con 0x7f48280c2d00 2024-03-28T15:10:37.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f4816ffd640 1 -- 172.21.15.186:0/3396155204 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c00aa30 con 0x7f48280c2d00 2024-03-28T15:10:37.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.143+0000 7f482cc81640 1 -- 172.21.15.186:0/3396155204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4828158d60 con 0x7f48280c2d00 2024-03-28T15:10:37.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.144+0000 7f482cc81640 1 -- 172.21.15.186:0/3396155204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4828069500 con 0x7f48280c2d00 2024-03-28T15:10:37.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.145+0000 7f4816ffd640 1 -- 172.21.15.186:0/3396155204 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f481c01b070 con 0x7f48280c2d00 2024-03-28T15:10:37.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.145+0000 7f4816ffd640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f480806c530 0x7f480806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.145+0000 7f4816ffd640 1 -- 172.21.15.186:0/3396155204 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f481c08af10 con 0x7f48280c2d00 2024-03-28T15:10:37.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.145+0000 7f482659c640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f480806c530 0x7f480806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.146+0000 7f482659c640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f480806c530 0x7f480806e9f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4820036000 tx=0x7f4820008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:37.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.149+0000 7f4816ffd640 1 -- 172.21.15.186:0/3396155204 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f481c058c30 con 0x7f48280c2d00 2024-03-28T15:10:37.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.326+0000 7f482cc81640 1 -- 172.21.15.186:0/3396155204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48280b1a00 con 0x7f48280c2d00 2024-03-28T15:10:37.327 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.327+0000 7f4816ffd640 1 -- 172.21.15.186:0/3396155204 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+17572 (secure 0 0 0) 0x7f481c05cf30 con 0x7f48280c2d00 2024-03-28T15:10:37.328 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:37.328 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":37,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:09:56.237674+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.191:6834/1992706523":"2024-03-29T15:08:31.220829+0000","172.21.15.191:6837/3037586083":"2024-03-29T15:09:56.237602+0000","172.21.15.191:6835/3037586083":"2024-03-29T15:09:56.237602+0000","172.21.15.191:6837/3828930138":"2024-03-29T15:06:06.202259+0000","172.21.15.191:6835/3828930138":"2024-03-29T15:06:06.202259+0000","172.21.15.191:6832/1992706523":"2024-03-29T15:08:31.220829+0000","172.21.15.191:6834/185536353":"2024-03-29T15:03:46.182756+0000","172.21.15.191:6832/185536353":"2024-03-29T15:03:46.182756+0000","172.21.15.191:6837/2813406930":"2024-03-29T15:02:11.169402+0000","172.21.15.191:6835/2813406930":"2024-03-29T15:02:11.169402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:10:37.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 -- 172.21.15.186:0/3396155204 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f480806c530 msgr2=0x7f480806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f480806c530 0x7f480806e9f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4820036000 tx=0x7f4820008030 comp rx=0 tx=0).stop 2024-03-28T15:10:37.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 -- 172.21.15.186:0/3396155204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48280c2d00 msgr2=0x7f482814cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48280c2d00 0x7f482814cc80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f481c00c7b0 tx=0x7f481c00cc70 comp rx=0 tx=0).stop 2024-03-28T15:10:37.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 -- 172.21.15.186:0/3396155204 shutdown_connections 2024-03-28T15:10:37.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f480806c530 0x7f480806e9f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4828151990 0x7f482814d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48280c2d00 0x7f482814cc80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 --2- 172.21.15.186:0/3396155204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48280b1e30 0x7f482814dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 -- 172.21.15.186:0/3396155204 >> 172.21.15.186:0/3396155204 conn(0x7f48280c8a20 msgr2=0x7f48280c7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:37.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 -- 172.21.15.186:0/3396155204 shutdown_connections 2024-03-28T15:10:37.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.329+0000 7f482cc81640 1 -- 172.21.15.186:0/3396155204 wait complete. 2024-03-28T15:10:37.341 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:01:18.661274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:01:29.512445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:01:30.388340+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 33, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:10:37.341 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:10:37.444 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.444+0000 7f09c4617640 1 Processor -- start 2024-03-28T15:10:37.445 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.444+0000 7f09c4617640 1 -- start start 2024-03-28T15:10:37.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c4617640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f09bc06db50 0x7f09bc06df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.446 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c4617640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09bc0c8600 con 0x7f09bc0c8110 2024-03-28T15:10:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c4617640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09bc157740 con 0x7f09bc151870 2024-03-28T15:10:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c4617640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f09bc158070 con 0x7f09bc06db50 2024-03-28T15:10:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c2bb4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f09bc06db50 0x7f09bc06df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c1bb2640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f09bc0c8110 0x7f09bc06ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:33494/0 (socket says 172.21.15.186:33494) 2024-03-28T15:10:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c1bb2640 1 -- 172.21.15.186:0/1628546160 learned_addr learned my addr 172.21.15.186:0/1628546160 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c23b3640 1 --1- 172.21.15.186:0/1628546160 >> v1:172.21.15.191:6789/0 conn(0x7f09bc151870 0x7f09bc0c7390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:38480/0 (socket says 172.21.15.186:38480) 2024-03-28T15:10:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c13b1640 1 -- 172.21.15.186:0/1628546160 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2743089637 0 0) 0x7f09bc157740 con 0x7f09bc151870 2024-03-28T15:10:37.447 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c2bb4640 1 -- 172.21.15.186:0/1628546160 >> v1:172.21.15.191:6789/0 conn(0x7f09bc151870 legacy=0x7f09bc0c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c2bb4640 1 -- 172.21.15.186:0/1628546160 >> v1:172.21.15.186:6789/0 conn(0x7f09bc0c8110 legacy=0x7f09bc06ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.445+0000 7f09c2bb4640 1 -- 172.21.15.186:0/1628546160 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09bc157a20 con 0x7f09bc06db50 2024-03-28T15:10:37.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.446+0000 7f09c2bb4640 1 --2- 172.21.15.186:0/1628546160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09bc06db50 0x7f09bc06df90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009d90 tx=0x7f09b402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f222c229cb111b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:37.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.446+0000 7f09c13b1640 1 -- 172.21.15.186:0/1628546160 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4039070 con 0x7f09bc06db50 2024-03-28T15:10:37.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.446+0000 7f09c13b1640 1 -- 172.21.15.186:0/1628546160 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b402cb40 con 0x7f09bc06db50 2024-03-28T15:10:37.448 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.446+0000 7f09c13b1640 1 -- 172.21.15.186:0/1628546160 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4034440 con 0x7f09bc06db50 2024-03-28T15:10:37.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.446+0000 7f09c4617640 1 -- 172.21.15.186:0/1628546160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09bc06db50 msgr2=0x7f09bc06df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.446+0000 7f09c4617640 1 --2- 172.21.15.186:0/1628546160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09bc06db50 0x7f09bc06df90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009d90 tx=0x7f09b402be10 comp rx=0 tx=0).stop 2024-03-28T15:10:37.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.447+0000 7f09c4617640 1 -- 172.21.15.186:0/1628546160 shutdown_connections 2024-03-28T15:10:37.452 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.447+0000 7f09c4617640 1 --2- 172.21.15.186:0/1628546160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09bc06db50 0x7f09bc06df90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.447+0000 7f09c4617640 1 -- 172.21.15.186:0/1628546160 >> 172.21.15.186:0/1628546160 conn(0x7f09bc06c010 msgr2=0x7f09bc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:37.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.447+0000 7f09c4617640 1 -- 172.21.15.186:0/1628546160 shutdown_connections 2024-03-28T15:10:37.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.447+0000 7f09c4617640 1 -- 172.21.15.186:0/1628546160 wait complete. 2024-03-28T15:10:37.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.448+0000 7f09c4617640 1 Processor -- start 2024-03-28T15:10:37.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.448+0000 7f09c4617640 1 -- start start 2024-03-28T15:10:37.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.448+0000 7f09c4617640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09bc06db50 0x7f09bc0c11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.448+0000 7f09c4617640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09bc0c8110 0x7f09bc0c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.453 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.448+0000 7f09c4617640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09bc151870 0x7f09bc0ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.448+0000 7f09c4617640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09bc06f2f0 con 0x7f09bc0c8110 2024-03-28T15:10:37.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c4617640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f09bc151c50 con 0x7f09bc151870 2024-03-28T15:10:37.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c4617640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f09bc1f4920 con 0x7f09bc06db50 2024-03-28T15:10:37.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c1bb2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09bc0c8110 0x7f09bc0c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c1bb2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09bc0c8110 0x7f09bc0c9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52866/0 (socket says 172.21.15.186:52866) 2024-03-28T15:10:37.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c1bb2640 1 -- 172.21.15.186:0/291706100 learned_addr learned my addr 172.21.15.186:0/291706100 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:37.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c23b3640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09bc06db50 0x7f09bc0c11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.454 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c2bb4640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09bc151870 0x7f09bc0ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c1bb2640 1 -- 172.21.15.186:0/291706100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09bc06db50 msgr2=0x7f09bc0c11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c1bb2640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09bc06db50 0x7f09bc0c11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c1bb2640 1 -- 172.21.15.186:0/291706100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09bc151870 msgr2=0x7f09bc0ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c1bb2640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09bc151870 0x7f09bc0ca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c1bb2640 1 -- 172.21.15.186:0/291706100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09bc1f1500 con 0x7f09bc0c8110 2024-03-28T15:10:37.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c1bb2640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09bc0c8110 0x7f09bc0c9d20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c780 tx=0x7f09ac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:37.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09b2ffd640 1 -- 172.21.15.186:0/291706100 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ac018070 con 0x7f09bc0c8110 2024-03-28T15:10:37.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09b2ffd640 1 -- 172.21.15.186:0/291706100 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09ac0072d0 con 0x7f09bc0c8110 2024-03-28T15:10:37.455 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c4617640 1 -- 172.21.15.186:0/291706100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09bc1f3d10 con 0x7f09bc0c8110 2024-03-28T15:10:37.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09b2ffd640 1 -- 172.21.15.186:0/291706100 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ac0047e0 con 0x7f09bc0c8110 2024-03-28T15:10:37.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.449+0000 7f09c4617640 1 -- 172.21.15.186:0/291706100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09bc1f42d0 con 0x7f09bc0c8110 2024-03-28T15:10:37.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.450+0000 7f09c4617640 1 -- 172.21.15.186:0/291706100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09bc069570 con 0x7f09bc0c8110 2024-03-28T15:10:37.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.451+0000 7f09b2ffd640 1 -- 172.21.15.186:0/291706100 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f09ac01b070 con 0x7f09bc0c8110 2024-03-28T15:10:37.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.451+0000 7f09b2ffd640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09a006c530 0x7f09a006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.451+0000 7f09b2ffd640 1 -- 172.21.15.186:0/291706100 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f09ac08ad80 con 0x7f09bc0c8110 2024-03-28T15:10:37.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.451+0000 7f09c23b3640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09a006c530 0x7f09a006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.456 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.451+0000 7f09c23b3640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09a006c530 0x7f09a006e9f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f09b8007fd0 tx=0x7f09b8007de0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:37.457 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.455+0000 7f09b2ffd640 1 -- 172.21.15.186:0/291706100 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f09ac058aa0 con 0x7f09bc0c8110 2024-03-28T15:10:37.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.629+0000 7f09c4617640 1 -- 172.21.15.186:0/291706100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f09bc0739e0 con 0x7f09bc0c8110 2024-03-28T15:10:37.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.630+0000 7f09b2ffd640 1 -- 172.21.15.186:0/291706100 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f09ac05cda0 con 0x7f09bc0c8110 2024-03-28T15:10:37.631 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 1 2024-03-28T15:10:37.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 -- 172.21.15.186:0/291706100 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09a006c530 msgr2=0x7f09a006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09a006c530 0x7f09a006e9f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f09b8007fd0 tx=0x7f09b8007de0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 -- 172.21.15.186:0/291706100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09bc0c8110 msgr2=0x7f09bc0c9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09bc0c8110 0x7f09bc0c9d20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c780 tx=0x7f09ac00cc40 comp rx=0 tx=0).stop 2024-03-28T15:10:37.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 -- 172.21.15.186:0/291706100 shutdown_connections 2024-03-28T15:10:37.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f09a006c530 0x7f09a006e9f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09bc151870 0x7f09bc0ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09bc0c8110 0x7f09bc0c9d20 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 --2- 172.21.15.186:0/291706100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09bc06db50 0x7f09bc0c11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 -- 172.21.15.186:0/291706100 >> 172.21.15.186:0/291706100 conn(0x7f09bc06c010 msgr2=0x7f09bc144eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 -- 172.21.15.186:0/291706100 shutdown_connections 2024-03-28T15:10:37.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.632+0000 7f09c4617640 1 -- 172.21.15.186:0/291706100 wait complete. 2024-03-28T15:10:37.644 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:10:37.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.747+0000 7f440a1d0640 1 Processor -- start 2024-03-28T15:10:37.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.748+0000 7f440a1d0640 1 -- start start 2024-03-28T15:10:37.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.748+0000 7f440a1d0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4404151870 0x7f44040c7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.748+0000 7f440a1d0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44040c8600 con 0x7f44040c8110 2024-03-28T15:10:37.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.748+0000 7f440a1d0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4404157740 con 0x7f440406db50 2024-03-28T15:10:37.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.748+0000 7f440a1d0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f4404158070 con 0x7f4404151870 2024-03-28T15:10:37.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.748+0000 7f44037fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4404151870 0x7f44040c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.748+0000 7f4402ffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f44040c8110 0x7f440406ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:33502/0 (socket says 172.21.15.186:33502) 2024-03-28T15:10:37.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.748+0000 7f4402ffd640 1 -- 172.21.15.186:0/608552857 learned_addr learned my addr 172.21.15.186:0/608552857 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:37.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.749+0000 7f44037fe640 1 -- 172.21.15.186:0/608552857 >> v1:172.21.15.191:6789/0 conn(0x7f440406db50 legacy=0x7f4404161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.749+0000 7f44037fe640 1 -- 172.21.15.186:0/608552857 >> v1:172.21.15.186:6789/0 conn(0x7f44040c8110 legacy=0x7f440406ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.749+0000 7f44027fc640 1 -- 172.21.15.186:0/608552857 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4022536904 0 0) 0x7f44040c8600 con 0x7f44040c8110 2024-03-28T15:10:37.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.749+0000 7f44037fe640 1 -- 172.21.15.186:0/608552857 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4404157a20 con 0x7f4404151870 2024-03-28T15:10:37.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.749+0000 7f44037fe640 1 --2- 172.21.15.186:0/608552857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4404151870 0x7f44040c7390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f43f40097a0 tx=0x7f43f402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e0fa6ad32d29f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:37.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.750+0000 7f44027fc640 1 -- 172.21.15.186:0/608552857 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f402f810 con 0x7f4404151870 2024-03-28T15:10:37.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.750+0000 7f44027fc640 1 -- 172.21.15.186:0/608552857 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f402f9b0 con 0x7f4404151870 2024-03-28T15:10:37.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.750+0000 7f44027fc640 1 -- 172.21.15.186:0/608552857 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f402fd80 con 0x7f4404151870 2024-03-28T15:10:37.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.750+0000 7f440a1d0640 1 -- 172.21.15.186:0/608552857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4404151870 msgr2=0x7f44040c7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.750+0000 7f440a1d0640 1 --2- 172.21.15.186:0/608552857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4404151870 0x7f44040c7390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f43f40097a0 tx=0x7f43f402b9f0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.750+0000 7f440a1d0640 1 -- 172.21.15.186:0/608552857 shutdown_connections 2024-03-28T15:10:37.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.750+0000 7f440a1d0640 1 --2- 172.21.15.186:0/608552857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4404151870 0x7f44040c7390 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.750+0000 7f440a1d0640 1 -- 172.21.15.186:0/608552857 >> 172.21.15.186:0/608552857 conn(0x7f440406c010 msgr2=0x7f44040cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:37.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.750+0000 7f440a1d0640 1 -- 172.21.15.186:0/608552857 shutdown_connections 2024-03-28T15:10:37.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.750+0000 7f440a1d0640 1 -- 172.21.15.186:0/608552857 wait complete. 2024-03-28T15:10:37.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.751+0000 7f440a1d0640 1 Processor -- start 2024-03-28T15:10:37.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f440a1d0640 1 -- start start 2024-03-28T15:10:37.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f440a1d0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f440406db50 0x7f44041500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f440a1d0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44040c8110 0x7f44040c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f440a1d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4404151870 0x7f440413f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f440a1d0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f44040c78d0 con 0x7f44040c8110 2024-03-28T15:10:37.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f440a1d0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4404151c50 con 0x7f440406db50 2024-03-28T15:10:37.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f440a1d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f440406f2f0 con 0x7f4404151870 2024-03-28T15:10:37.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f4402ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44040c8110 0x7f44040c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f44037fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f440406db50 0x7f44041500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f4402ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44040c8110 0x7f44040c9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52880/0 (socket says 172.21.15.186:52880) 2024-03-28T15:10:37.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f4402ffd640 1 -- 172.21.15.186:0/1687374727 learned_addr learned my addr 172.21.15.186:0/1687374727 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:37.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.752+0000 7f4403fff640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4404151870 0x7f440413f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.753+0000 7f44037fe640 1 -- 172.21.15.186:0/1687374727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4404151870 msgr2=0x7f440413f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.753+0000 7f44037fe640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4404151870 0x7f440413f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.753+0000 7f44037fe640 1 -- 172.21.15.186:0/1687374727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44040c8110 msgr2=0x7f44040c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.753+0000 7f44037fe640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44040c8110 0x7f44040c9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.753+0000 7f44037fe640 1 -- 172.21.15.186:0/1687374727 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44041505e0 con 0x7f440406db50 2024-03-28T15:10:37.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.753+0000 7f44037fe640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f440406db50 0x7f44041500a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f43f4033000 tx=0x7f43f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:37.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.753+0000 7f43ebfff640 1 -- 172.21.15.186:0/1687374727 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4046070 con 0x7f440406db50 2024-03-28T15:10:37.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.753+0000 7f43ebfff640 1 -- 172.21.15.186:0/1687374727 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f4037c60 con 0x7f440406db50 2024-03-28T15:10:37.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.753+0000 7f440a1d0640 1 -- 172.21.15.186:0/1687374727 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44040c5ed0 con 0x7f440406db50 2024-03-28T15:10:37.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.753+0000 7f43ebfff640 1 -- 172.21.15.186:0/1687374727 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f402fe60 con 0x7f440406db50 2024-03-28T15:10:37.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.753+0000 7f440a1d0640 1 -- 172.21.15.186:0/1687374727 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4404146950 con 0x7f440406db50 2024-03-28T15:10:37.759 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.754+0000 7f440a1d0640 1 -- 172.21.15.186:0/1687374727 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4404069570 con 0x7f440406db50 2024-03-28T15:10:37.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.754+0000 7f43ebfff640 1 -- 172.21.15.186:0/1687374727 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f43f402f840 con 0x7f440406db50 2024-03-28T15:10:37.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.755+0000 7f43ebfff640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f43f006c530 0x7f43f006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:37.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.755+0000 7f43ebfff640 1 -- 172.21.15.186:0/1687374727 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f43f40ad170 con 0x7f440406db50 2024-03-28T15:10:37.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.755+0000 7f4402ffd640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f43f006c530 0x7f43f006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:37.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.755+0000 7f4402ffd640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f43f006c530 0x7f43f006e9f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f43f8007fd0 tx=0x7f43f8007f60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:37.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.764+0000 7f43ebfff640 1 -- 172.21.15.186:0/1687374727 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f43f403e070 con 0x7f440406db50 2024-03-28T15:10:37.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.935+0000 7f440a1d0640 1 -- 172.21.15.186:0/1687374727 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f44040cbf20 con 0x7f440406db50 2024-03-28T15:10:37.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.936+0000 7f43ebfff640 1 -- 172.21.15.186:0/1687374727 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) ==== 104+0+11 (secure 0 0 0) 0x7f43f4033030 con 0x7f440406db50 2024-03-28T15:10:37.937 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 32 2024-03-28T15:10:37.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 -- 172.21.15.186:0/1687374727 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f43f006c530 msgr2=0x7f43f006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f43f006c530 0x7f43f006e9f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f43f8007fd0 tx=0x7f43f8007f60 comp rx=0 tx=0).stop 2024-03-28T15:10:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 -- 172.21.15.186:0/1687374727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f440406db50 msgr2=0x7f44041500a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f440406db50 0x7f44041500a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f43f4033000 tx=0x7f43f40046c0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 -- 172.21.15.186:0/1687374727 shutdown_connections 2024-03-28T15:10:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f43f006c530 0x7f43f006e9f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4404151870 0x7f440413f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44040c8110 0x7f44040c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 --2- 172.21.15.186:0/1687374727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f440406db50 0x7f44041500a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:37.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 -- 172.21.15.186:0/1687374727 >> 172.21.15.186:0/1687374727 conn(0x7f440406c010 msgr2=0x7f44041419c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:37.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 -- 172.21.15.186:0/1687374727 shutdown_connections 2024-03-28T15:10:37.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:37.938+0000 7f440a1d0640 1 -- 172.21.15.186:0/1687374727 wait complete. 2024-03-28T15:10:37.950 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:10:38.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.055+0000 7fbf43f7a640 1 Processor -- start 2024-03-28T15:10:38.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.056+0000 7fbf43f7a640 1 -- start start 2024-03-28T15:10:38.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.056+0000 7fbf43f7a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06f230 0x7fbf3c06df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:38.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.056+0000 7fbf43f7a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbf3c159180 0x7fbf3c1525a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:38.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.056+0000 7fbf43f7a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf3c072370 con 0x7fbf3c06ee50 2024-03-28T15:10:38.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.056+0000 7fbf43f7a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fbf3c0c87b0 con 0x7fbf3c159180 2024-03-28T15:10:38.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.056+0000 7fbf43f7a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fbf3c071ee0 con 0x7fbf3c06f230 2024-03-28T15:10:38.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.057+0000 7fbf41d16640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fbf3c06ee50 0x7fbf3c06c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:33516/0 (socket says 172.21.15.186:33516) 2024-03-28T15:10:38.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.057+0000 7fbf41d16640 1 -- 172.21.15.186:0/3078063639 learned_addr learned my addr 172.21.15.186:0/3078063639 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:38.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.057+0000 7fbf41515640 1 --2- 172.21.15.186:0/3078063639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06f230 0x7fbf3c06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:38.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.057+0000 7fbf42517640 1 --2- 172.21.15.186:0/3078063639 >> v2:172.21.15.191:3300/0 conn(0x7fbf3c159180 0x7fbf3c1525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:38.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.057+0000 7fbf40d14640 1 -- 172.21.15.186:0/3078063639 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1538502953 0 0) 0x7fbf3c072370 con 0x7fbf3c06ee50 2024-03-28T15:10:38.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.057+0000 7fbf40d14640 1 -- 172.21.15.186:0/3078063639 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf28003520 con 0x7fbf3c06ee50 2024-03-28T15:10:38.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.057+0000 7fbf42517640 1 -- 172.21.15.186:0/3078063639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06f230 msgr2=0x7fbf3c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:38.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.057+0000 7fbf42517640 1 --2- 172.21.15.186:0/3078063639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06f230 0x7fbf3c06df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.057+0000 7fbf42517640 1 -- 172.21.15.186:0/3078063639 >> v1:172.21.15.186:6789/0 conn(0x7fbf3c06ee50 legacy=0x7fbf3c06c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.057+0000 7fbf42517640 1 -- 172.21.15.186:0/3078063639 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3c06e550 con 0x7fbf3c159180 2024-03-28T15:10:38.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.057+0000 7fbf41515640 1 --2- 172.21.15.186:0/3078063639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06f230 0x7fbf3c06df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:38.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.058+0000 7fbf42517640 1 --2- 172.21.15.186:0/3078063639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf3c159180 0x7fbf3c1525a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009d90 tx=0x7fbf3402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eab87dcc41bce68 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:38.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.058+0000 7fbf40d14640 1 -- 172.21.15.186:0/3078063639 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34039070 con 0x7fbf3c159180 2024-03-28T15:10:38.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.058+0000 7fbf40d14640 1 -- 172.21.15.186:0/3078063639 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf3402cc50 con 0x7fbf3c159180 2024-03-28T15:10:38.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.058+0000 7fbf40d14640 1 -- 172.21.15.186:0/3078063639 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34034440 con 0x7fbf3c159180 2024-03-28T15:10:38.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.058+0000 7fbf43f7a640 1 -- 172.21.15.186:0/3078063639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf3c159180 msgr2=0x7fbf3c1525a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.058+0000 7fbf43f7a640 1 --2- 172.21.15.186:0/3078063639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf3c159180 0x7fbf3c1525a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009d90 tx=0x7fbf3402bea0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.058+0000 7fbf43f7a640 1 -- 172.21.15.186:0/3078063639 shutdown_connections 2024-03-28T15:10:38.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.058+0000 7fbf43f7a640 1 --2- 172.21.15.186:0/3078063639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf3c159180 0x7fbf3c1525a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.058+0000 7fbf43f7a640 1 --2- 172.21.15.186:0/3078063639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06f230 0x7fbf3c06df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.058+0000 7fbf43f7a640 1 -- 172.21.15.186:0/3078063639 >> 172.21.15.186:0/3078063639 conn(0x7fbf3c06c010 msgr2=0x7fbf3c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:38.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.059+0000 7fbf43f7a640 1 -- 172.21.15.186:0/3078063639 shutdown_connections 2024-03-28T15:10:38.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.059+0000 7fbf43f7a640 1 -- 172.21.15.186:0/3078063639 wait complete. 2024-03-28T15:10:38.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.060+0000 7fbf43f7a640 1 Processor -- start 2024-03-28T15:10:38.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.060+0000 7fbf43f7a640 1 -- start start 2024-03-28T15:10:38.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.060+0000 7fbf43f7a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06ee50 0x7fbf3c14afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:38.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf43f7a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf3c06f230 0x7fbf3c153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:38.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf43f7a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf3c159180 0x7fbf3c1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:38.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf43f7a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c071890 con 0x7fbf3c159180 2024-03-28T15:10:38.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf43f7a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c0715b0 con 0x7fbf3c06f230 2024-03-28T15:10:38.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf43f7a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fbf3c071b70 con 0x7fbf3c06ee50 2024-03-28T15:10:38.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf42517640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf3c159180 0x7fbf3c1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:38.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf41d16640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06ee50 0x7fbf3c14afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:38.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf42517640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf3c159180 0x7fbf3c1553b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52892/0 (socket says 172.21.15.186:52892) 2024-03-28T15:10:38.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf41515640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf3c06f230 0x7fbf3c153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:38.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf41d16640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06ee50 0x7fbf3c14afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:51690/0 (socket says 172.21.15.186:51690) 2024-03-28T15:10:38.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf42517640 1 -- 172.21.15.186:0/846482628 learned_addr learned my addr 172.21.15.186:0/846482628 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:38.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf42517640 1 -- 172.21.15.186:0/846482628 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06ee50 msgr2=0x7fbf3c14afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf42517640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06ee50 0x7fbf3c14afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf42517640 1 -- 172.21.15.186:0/846482628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf3c06f230 msgr2=0x7fbf3c153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf42517640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf3c06f230 0x7fbf3c153770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf42517640 1 -- 172.21.15.186:0/846482628 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3c1e5250 con 0x7fbf3c159180 2024-03-28T15:10:38.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf42517640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf3c159180 0x7fbf3c1553b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009d90 tx=0x7fbf34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:38.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf327fc640 1 -- 172.21.15.186:0/846482628 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34042030 con 0x7fbf3c159180 2024-03-28T15:10:38.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf327fc640 1 -- 172.21.15.186:0/846482628 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf3403a040 con 0x7fbf3c159180 2024-03-28T15:10:38.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf43f7a640 1 -- 172.21.15.186:0/846482628 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf3c1e5530 con 0x7fbf3c159180 2024-03-28T15:10:38.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf327fc640 1 -- 172.21.15.186:0/846482628 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34039070 con 0x7fbf3c159180 2024-03-28T15:10:38.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.061+0000 7fbf43f7a640 1 -- 172.21.15.186:0/846482628 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf3c158800 con 0x7fbf3c159180 2024-03-28T15:10:38.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.062+0000 7fbf43f7a640 1 -- 172.21.15.186:0/846482628 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf3c069570 con 0x7fbf3c159180 2024-03-28T15:10:38.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.063+0000 7fbf327fc640 1 -- 172.21.15.186:0/846482628 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbf34045070 con 0x7fbf3c159180 2024-03-28T15:10:38.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.063+0000 7fbf327fc640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf2006c560 0x7fbf2006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:38.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.063+0000 7fbf327fc640 1 -- 172.21.15.186:0/846482628 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fbf340ad4f0 con 0x7fbf3c159180 2024-03-28T15:10:38.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.063+0000 7fbf41d16640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf2006c560 0x7fbf2006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:38.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.064+0000 7fbf41d16640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf2006c560 0x7fbf2006ea20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbf38005880 tx=0x7fbf38009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:38.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.068+0000 7fbf327fc640 1 -- 172.21.15.186:0/846482628 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fbf3404c070 con 0x7fbf3c159180 2024-03-28T15:10:38.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.260+0000 7fbf43f7a640 1 -- 172.21.15.186:0/846482628 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbf3c14bd10 con 0x7fbf3c159180 2024-03-28T15:10:38.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.260+0000 7fbf327fc640 1 -- 172.21.15.186:0/846482628 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) ==== 100+0+11 (secure 0 0 0) 0x7fbf3407b2d0 con 0x7fbf3c159180 2024-03-28T15:10:38.261 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 64 2024-03-28T15:10:38.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 -- 172.21.15.186:0/846482628 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf2006c560 msgr2=0x7fbf2006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf2006c560 0x7fbf2006ea20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbf38005880 tx=0x7fbf38009040 comp rx=0 tx=0).stop 2024-03-28T15:10:38.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 -- 172.21.15.186:0/846482628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf3c159180 msgr2=0x7fbf3c1553b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf3c159180 0x7fbf3c1553b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009d90 tx=0x7fbf34004a40 comp rx=0 tx=0).stop 2024-03-28T15:10:38.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 -- 172.21.15.186:0/846482628 shutdown_connections 2024-03-28T15:10:38.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fbf2006c560 0x7fbf2006ea20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf3c159180 0x7fbf3c1553b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf3c06f230 0x7fbf3c153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 --2- 172.21.15.186:0/846482628 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf3c06ee50 0x7fbf3c14afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 -- 172.21.15.186:0/846482628 >> 172.21.15.186:0/846482628 conn(0x7fbf3c06c010 msgr2=0x7fbf3c0ca640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:38.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 -- 172.21.15.186:0/846482628 shutdown_connections 2024-03-28T15:10:38.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.262+0000 7fbf43f7a640 1 -- 172.21.15.186:0/846482628 wait complete. 2024-03-28T15:10:38.274 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T15:10:38.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.377+0000 7fdb363f7640 1 Processor -- start 2024-03-28T15:10:38.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.377+0000 7fdb363f7640 1 -- start start 2024-03-28T15:10:38.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.378+0000 7fdb363f7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb3015a7c0 0x7fdb300ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:38.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.378+0000 7fdb363f7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb300cb6e0 con 0x7fdb30154b60 2024-03-28T15:10:38.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.378+0000 7fdb363f7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb3015b760 con 0x7fdb30071a80 2024-03-28T15:10:38.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.378+0000 7fdb363f7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdb30072820 con 0x7fdb3015a7c0 2024-03-28T15:10:38.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.378+0000 7fdb2f7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb3015a7c0 0x7fdb300ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:38.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.378+0000 7fdb2ffff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fdb30154b60 0x7fdb300cafd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:33520/0 (socket says 172.21.15.186:33520) 2024-03-28T15:10:38.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.378+0000 7fdb2ffff640 1 -- 172.21.15.186:0/1788530423 learned_addr learned my addr 172.21.15.186:0/1788530423 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.379+0000 7fdb2f7fe640 1 -- 172.21.15.186:0/1788530423 >> v1:172.21.15.191:6789/0 conn(0x7fdb30071a80 legacy=0x7fdb30165140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.379+0000 7fdb2f7fe640 1 -- 172.21.15.186:0/1788530423 >> v1:172.21.15.186:6789/0 conn(0x7fdb30154b60 legacy=0x7fdb300cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.379+0000 7fdb2f7fe640 1 -- 172.21.15.186:0/1788530423 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3015aec0 con 0x7fdb3015a7c0 2024-03-28T15:10:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.379+0000 7fdb2f7fe640 1 --2- 172.21.15.186:0/1788530423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb3015a7c0 0x7fdb300ccf10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdb240097a0 tx=0x7fdb2402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c196f3edb4786237 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.379+0000 7fdb2effd640 1 -- 172.21.15.186:0/1788530423 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2402f810 con 0x7fdb3015a7c0 2024-03-28T15:10:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.380+0000 7fdb2effd640 1 -- 172.21.15.186:0/1788530423 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb2402f9b0 con 0x7fdb3015a7c0 2024-03-28T15:10:38.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.380+0000 7fdb2effd640 1 -- 172.21.15.186:0/1788530423 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2402fd80 con 0x7fdb3015a7c0 2024-03-28T15:10:38.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.380+0000 7fdb363f7640 1 -- 172.21.15.186:0/1788530423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb3015a7c0 msgr2=0x7fdb300ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.380+0000 7fdb363f7640 1 --2- 172.21.15.186:0/1788530423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb3015a7c0 0x7fdb300ccf10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdb240097a0 tx=0x7fdb2402b9f0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.380+0000 7fdb363f7640 1 -- 172.21.15.186:0/1788530423 shutdown_connections 2024-03-28T15:10:38.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.380+0000 7fdb363f7640 1 --2- 172.21.15.186:0/1788530423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb3015a7c0 0x7fdb300ccf10 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.380+0000 7fdb363f7640 1 -- 172.21.15.186:0/1788530423 >> 172.21.15.186:0/1788530423 conn(0x7fdb30073b80 msgr2=0x7fdb3006b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:38.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.380+0000 7fdb363f7640 1 -- 172.21.15.186:0/1788530423 shutdown_connections 2024-03-28T15:10:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.380+0000 7fdb363f7640 1 -- 172.21.15.186:0/1788530423 wait complete. 2024-03-28T15:10:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.381+0000 7fdb363f7640 1 Processor -- start 2024-03-28T15:10:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.381+0000 7fdb363f7640 1 -- start start 2024-03-28T15:10:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb363f7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdb30071a80 0x7fdb300b5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb363f7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb30154b60 0x7fdb300b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb363f7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb3015a7c0 0x7fdb3015c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb363f7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdb300cc710 con 0x7fdb30154b60 2024-03-28T15:10:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb363f7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3015b480 con 0x7fdb3015a7c0 2024-03-28T15:10:38.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb363f7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fdb3015abe0 con 0x7fdb30071a80 2024-03-28T15:10:38.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb2f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb30154b60 0x7fdb300b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:38.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb2f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb30154b60 0x7fdb300b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52900/0 (socket says 172.21.15.186:52900) 2024-03-28T15:10:38.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb2f7fe640 1 -- 172.21.15.186:0/3498012128 learned_addr learned my addr 172.21.15.186:0/3498012128 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:38.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb2ffff640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdb30071a80 0x7fdb300b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:38.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb34994640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb3015a7c0 0x7fdb3015c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:38.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb2f7fe640 1 -- 172.21.15.186:0/3498012128 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdb30071a80 msgr2=0x7fdb300b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb2f7fe640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdb30071a80 0x7fdb300b5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb2f7fe640 1 -- 172.21.15.186:0/3498012128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb3015a7c0 msgr2=0x7fdb3015c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb2f7fe640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb3015a7c0 0x7fdb3015c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.382+0000 7fdb2f7fe640 1 -- 172.21.15.186:0/3498012128 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3014ab80 con 0x7fdb30154b60 2024-03-28T15:10:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.383+0000 7fdb2f7fe640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb30154b60 0x7fdb300b5a60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdb2402c000 tx=0x7fdb24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.383+0000 7fdb2cff9640 1 -- 172.21.15.186:0/3498012128 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb24047070 con 0x7fdb30154b60 2024-03-28T15:10:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.383+0000 7fdb2cff9640 1 -- 172.21.15.186:0/3498012128 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb24037c60 con 0x7fdb30154b60 2024-03-28T15:10:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.383+0000 7fdb363f7640 1 -- 172.21.15.186:0/3498012128 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb3014ae40 con 0x7fdb30154b60 2024-03-28T15:10:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.383+0000 7fdb2cff9640 1 -- 172.21.15.186:0/3498012128 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2402fe60 con 0x7fdb30154b60 2024-03-28T15:10:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.383+0000 7fdb363f7640 1 -- 172.21.15.186:0/3498012128 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb300c9490 con 0x7fdb30154b60 2024-03-28T15:10:38.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.384+0000 7fdb363f7640 1 -- 172.21.15.186:0/3498012128 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb300686d0 con 0x7fdb30154b60 2024-03-28T15:10:38.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.384+0000 7fdb2cff9640 1 -- 172.21.15.186:0/3498012128 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdb2404a070 con 0x7fdb30154b60 2024-03-28T15:10:38.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.385+0000 7fdb2cff9640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdb1c06c530 0x7fdb1c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:38.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.385+0000 7fdb2cff9640 1 -- 172.21.15.186:0/3498012128 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fdb240ad5e0 con 0x7fdb30154b60 2024-03-28T15:10:38.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.385+0000 7fdb2ffff640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdb1c06c530 0x7fdb1c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:38.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.385+0000 7fdb2ffff640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdb1c06c530 0x7fdb1c06e9f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdb20005880 tx=0x7fdb20009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:38.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.389+0000 7fdb2cff9640 1 -- 172.21.15.186:0/3498012128 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fdb24033030 con 0x7fdb30154b60 2024-03-28T15:10:38.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.575+0000 7fdb363f7640 1 -- 172.21.15.186:0/3498012128 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdb300cb910 con 0x7fdb30154b60 2024-03-28T15:10:38.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.575+0000 7fdb2cff9640 1 -- 172.21.15.186:0/3498012128 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v37) ==== 81+0+618 (secure 0 0 0) 0x7fdb2407b390 con 0x7fdb30154b60 2024-03-28T15:10:38.576 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6834/1992706523 2024-03-29T15:08:31.220829+0000 2024-03-28T15:10:38.577 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6837/3037586083 2024-03-29T15:09:56.237602+0000 2024-03-28T15:10:38.577 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6835/3037586083 2024-03-29T15:09:56.237602+0000 2024-03-28T15:10:38.577 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6837/3828930138 2024-03-29T15:06:06.202259+0000 2024-03-28T15:10:38.577 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6835/3828930138 2024-03-29T15:06:06.202259+0000 2024-03-28T15:10:38.577 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6832/1992706523 2024-03-29T15:08:31.220829+0000 2024-03-28T15:10:38.577 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6834/185536353 2024-03-29T15:03:46.182756+0000 2024-03-28T15:10:38.577 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6832/185536353 2024-03-29T15:03:46.182756+0000 2024-03-28T15:10:38.578 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6837/2813406930 2024-03-29T15:02:11.169402+0000 2024-03-28T15:10:38.578 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.191:6835/2813406930 2024-03-29T15:02:11.169402+0000 2024-03-28T15:10:38.579 INFO:teuthology.orchestra.run.smithi186.stderr:listed 10 entries 2024-03-28T15:10:38.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.577+0000 7fdb363f7640 1 -- 172.21.15.186:0/3498012128 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdb1c06c530 msgr2=0x7fdb1c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.577+0000 7fdb363f7640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdb1c06c530 0x7fdb1c06e9f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdb20005880 tx=0x7fdb20009040 comp rx=0 tx=0).stop 2024-03-28T15:10:38.579 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.577+0000 7fdb363f7640 1 -- 172.21.15.186:0/3498012128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb30154b60 msgr2=0x7fdb300b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:38.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.577+0000 7fdb363f7640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb30154b60 0x7fdb300b5a60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdb2402c000 tx=0x7fdb24004730 comp rx=0 tx=0).stop 2024-03-28T15:10:38.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.578+0000 7fdb363f7640 1 -- 172.21.15.186:0/3498012128 shutdown_connections 2024-03-28T15:10:38.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.578+0000 7fdb363f7640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fdb1c06c530 0x7fdb1c06e9f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.578+0000 7fdb363f7640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb3015a7c0 0x7fdb3015c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.578+0000 7fdb363f7640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb30154b60 0x7fdb300b5a60 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.578+0000 7fdb363f7640 1 --2- 172.21.15.186:0/3498012128 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdb30071a80 0x7fdb300b5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:38.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.578+0000 7fdb363f7640 1 -- 172.21.15.186:0/3498012128 >> 172.21.15.186:0/3498012128 conn(0x7fdb30073b80 msgr2=0x7fdb300cbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:38.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.578+0000 7fdb363f7640 1 -- 172.21.15.186:0/3498012128 shutdown_connections 2024-03-28T15:10:38.581 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:38.578+0000 7fdb363f7640 1 -- 172.21.15.186:0/3498012128 wait complete. 2024-03-28T15:10:38.589 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi186.front.sepia.ceph.com... 2024-03-28T15:10:38.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:10:38.590 DEBUG:teuthology.orchestra.run.smithi186:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-28T15:10:38.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi186.stderr:ceph-fuse[46497]: fuse finished with error 0 and tester_r 0 2024-03-28T15:10:38.668 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:10:44.770 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T15:10:44.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:10:44.771 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T15:10:44.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:10:44.790 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2024-03-28T15:10:44.848 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T15:10:44.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:10:44.850 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T15:10:44.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:10:44.921 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link delete ceph-brx 2024-03-28T15:10:45.022 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-28T15:10:45.032 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-28T15:10:45.032 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:10:45.137 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.137+0000 7f5c9a0b5640 1 Processor -- start 2024-03-28T15:10:45.137 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.137+0000 7f5c9a0b5640 1 -- start start 2024-03-28T15:10:45.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.137+0000 7f5c9a0b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c94158120 0x7f5c940c4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.137+0000 7f5c9a0b5640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5c940c30b0 0x7f5c94067280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.137+0000 7f5c9a0b5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c940c8eb0 con 0x7f5c940677c0 2024-03-28T15:10:45.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.137+0000 7f5c9a0b5640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f5c940ce020 con 0x7f5c940c30b0 2024-03-28T15:10:45.140 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.137+0000 7f5c9a0b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5c940ce270 con 0x7f5c94158120 2024-03-28T15:10:45.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.138+0000 7f5c92ffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5c940c30b0 0x7f5c94067280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.138+0000 7f5c937fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c94158120 0x7f5c940c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.138+0000 7f5c93fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f5c940677c0 0x7f5c9415ddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:57508/0 (socket says 172.21.15.186:57508) 2024-03-28T15:10:45.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.138+0000 7f5c93fff640 1 -- 172.21.15.186:0/1861890874 learned_addr learned my addr 172.21.15.186:0/1861890874 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:45.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.138+0000 7f5c937fe640 1 --2- 172.21.15.186:0/1861890874 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c94158120 0x7f5c940c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:56186/0 (socket says 172.21.15.186:56186) 2024-03-28T15:10:45.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.138+0000 7f5c92ffd640 1 --2- 172.21.15.186:0/1861890874 >> v2:172.21.15.191:3300/0 conn(0x7f5c940c30b0 0x7f5c94067280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:43144/0 (socket says 172.21.15.186:43144) 2024-03-28T15:10:45.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.138+0000 7f5c937fe640 1 -- 172.21.15.186:0/1861890874 >> v2:172.21.15.191:3300/0 conn(0x7f5c940c30b0 msgr2=0x7f5c94067280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:45.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.138+0000 7f5c937fe640 1 --2- 172.21.15.186:0/1861890874 >> v2:172.21.15.191:3300/0 conn(0x7f5c940c30b0 0x7f5c94067280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.138+0000 7f5c937fe640 1 -- 172.21.15.186:0/1861890874 >> v1:172.21.15.191:6789/0 conn(0x7f5c940677c0 legacy=0x7f5c9415ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.138+0000 7f5c937fe640 1 -- 172.21.15.186:0/1861890874 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c940ce550 con 0x7f5c94158120 2024-03-28T15:10:45.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.139+0000 7f5c937fe640 1 --2- 172.21.15.186:0/1861890874 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c94158120 0x7f5c940c4440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5c880097a0 tx=0x7f5c8802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4666a828e1f5e0d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:45.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.139+0000 7f5c927fc640 1 -- 172.21.15.186:0/1861890874 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c88035020 con 0x7f5c94158120 2024-03-28T15:10:45.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.139+0000 7f5c927fc640 1 -- 172.21.15.186:0/1861890874 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c88031830 con 0x7f5c94158120 2024-03-28T15:10:45.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.139+0000 7f5c927fc640 1 -- 172.21.15.186:0/1861890874 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c88031c00 con 0x7f5c94158120 2024-03-28T15:10:45.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.139+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/1861890874 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c94158120 msgr2=0x7f5c940c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.139+0000 7f5c9a0b5640 1 --2- 172.21.15.186:0/1861890874 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c94158120 0x7f5c940c4440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5c880097a0 tx=0x7f5c8802b8d0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.139+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/1861890874 shutdown_connections 2024-03-28T15:10:45.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.139+0000 7f5c9a0b5640 1 --2- 172.21.15.186:0/1861890874 >> v2:172.21.15.191:3300/0 conn(0x7f5c940c30b0 0x7f5c94067280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.139+0000 7f5c9a0b5640 1 --2- 172.21.15.186:0/1861890874 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c94158120 0x7f5c940c4440 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.139+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/1861890874 >> 172.21.15.186:0/1861890874 conn(0x7f5c9406fc80 msgr2=0x7f5c940734c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:45.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.139+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/1861890874 shutdown_connections 2024-03-28T15:10:45.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.140+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/1861890874 wait complete. 2024-03-28T15:10:45.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.141+0000 7f5c9a0b5640 1 Processor -- start 2024-03-28T15:10:45.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.141+0000 7f5c9a0b5640 1 -- start start 2024-03-28T15:10:45.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.141+0000 7f5c9a0b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c940677c0 0x7f5c9406cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.141+0000 7f5c9a0b5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c940c30b0 0x7f5c9406c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.141+0000 7f5c9a0b5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94158120 0x7f5c9415e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.141+0000 7f5c9a0b5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c940c5a80 con 0x7f5c940c30b0 2024-03-28T15:10:45.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.141+0000 7f5c9a0b5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5c940ceb10 con 0x7f5c94158120 2024-03-28T15:10:45.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c9a0b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5c940c8a90 con 0x7f5c940677c0 2024-03-28T15:10:45.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c92ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c940c30b0 0x7f5c9406c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c92ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c940c30b0 0x7f5c9406c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36638/0 (socket says 172.21.15.186:36638) 2024-03-28T15:10:45.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c92ffd640 1 -- 172.21.15.186:0/873343335 learned_addr learned my addr 172.21.15.186:0/873343335 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:45.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c937fe640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c940677c0 0x7f5c9406cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c93fff640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94158120 0x7f5c9415e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c937fe640 1 -- 172.21.15.186:0/873343335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94158120 msgr2=0x7f5c9415e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c937fe640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94158120 0x7f5c9415e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c937fe640 1 -- 172.21.15.186:0/873343335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c940c30b0 msgr2=0x7f5c9406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c937fe640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c940c30b0 0x7f5c9406c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c937fe640 1 -- 172.21.15.186:0/873343335 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c940688d0 con 0x7f5c940677c0 2024-03-28T15:10:45.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c92ffd640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c940c30b0 0x7f5c9406c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:45.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c937fe640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c940677c0 0x7f5c9406cc30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5c88033000 tx=0x7f5c880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:45.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c7bfff640 1 -- 172.21.15.186:0/873343335 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8803b8c0 con 0x7f5c940677c0 2024-03-28T15:10:45.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c7bfff640 1 -- 172.21.15.186:0/873343335 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c8803ba60 con 0x7f5c940677c0 2024-03-28T15:10:45.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c7bfff640 1 -- 172.21.15.186:0/873343335 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8803be50 con 0x7f5c940677c0 2024-03-28T15:10:45.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/873343335 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c9415ebf0 con 0x7f5c940677c0 2024-03-28T15:10:45.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.142+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/873343335 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c9406b7e0 con 0x7f5c940677c0 2024-03-28T15:10:45.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.143+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/873343335 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c940cc270 con 0x7f5c940677c0 2024-03-28T15:10:45.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.144+0000 7f5c7bfff640 1 -- 172.21.15.186:0/873343335 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5c8804c060 con 0x7f5c940677c0 2024-03-28T15:10:45.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.144+0000 7f5c7bfff640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5c8006c700 0x7f5c8006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.144+0000 7f5c7bfff640 1 -- 172.21.15.186:0/873343335 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f5c880ad550 con 0x7f5c940677c0 2024-03-28T15:10:45.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.144+0000 7f5c92ffd640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5c8006c700 0x7f5c8006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.145+0000 7f5c92ffd640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5c8006c700 0x7f5c8006ebc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5c9415bfe0 tx=0x7f5c84009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:45.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.153+0000 7f5c7bfff640 1 -- 172.21.15.186:0/873343335 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f5c940cc270 con 0x7f5c940677c0 2024-03-28T15:10:45.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.325+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/873343335 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5c94069500 con 0x7f5c8006c700 2024-03-28T15:10:45.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.331+0000 7f5c7bfff640 1 -- 172.21.15.186:0/873343335 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+256109 (secure 0 0 0) 0x7f5c94069500 con 0x7f5c8006c700 2024-03-28T15:10:45.335 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:45.336 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:10:45.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/873343335 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5c8006c700 msgr2=0x7f5c8006ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5c8006c700 0x7f5c8006ebc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5c9415bfe0 tx=0x7f5c84009190 comp rx=0 tx=0).stop 2024-03-28T15:10:45.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/873343335 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c940677c0 msgr2=0x7f5c9406cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c940677c0 0x7f5c9406cc30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5c88033000 tx=0x7f5c880046c0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/873343335 shutdown_connections 2024-03-28T15:10:45.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f5c8006c700 0x7f5c8006ebc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94158120 0x7f5c9415e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c940c30b0 0x7f5c9406c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 --2- 172.21.15.186:0/873343335 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c940677c0 0x7f5c9406cc30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/873343335 >> 172.21.15.186:0/873343335 conn(0x7f5c9406fc80 msgr2=0x7f5c94154570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:45.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/873343335 shutdown_connections 2024-03-28T15:10:45.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.334+0000 7f5c9a0b5640 1 -- 172.21.15.186:0/873343335 wait complete. 2024-03-28T15:10:45.346 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-03-28T15:10:44.750496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1123802199,"num_objects":10721,"num_object_clones":0,"num_object_copies":21442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10721,"num_whiteouts":0,"num_read":31288,"num_read_kb":1091324,"num_write":28249,"num_write_kb":1250558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28537,"ondisk_log_size":28537,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2684488,"kb_used_data":2250568,"kb_used_omap":12,"kb_used_meta":433843,"kb_avail":752290232,"statfs":{"total":773094113280,"available":770345197568,"internally_reserved":0,"allocated":2304581632,"data_stored":2247448000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":444255793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26311273,"num_objects":1458,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":6,"num_read_kb":908,"num_write":1684,"num_write_kb":27163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003096"},"pg_stats":[{"pgid":"3.3f","version":"37'335","reported_seq":3248,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.950296+0000","last_change":"2024-03-28T15:04:56.745680+0000","last_active":"2024-03-28T15:10:38.950296+0000","last_peered":"2024-03-28T15:10:38.950296+0000","last_clean":"2024-03-28T15:10:38.950296+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:10:38.950296+0000","last_undegraded":"2024-03-28T15:10:38.950296+0000","last_fullsized":"2024-03-28T15:10:38.950296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:21:41.971527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024885649999999999,"stat_sum":{"num_bytes":26509954,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":839,"num_read_kb":12103,"num_write":327,"num_write_kb":26054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"37'316","reported_seq":2406,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.930520+0000","last_change":"2024-03-28T15:04:56.294141+0000","last_active":"2024-03-28T15:10:38.930520+0000","last_peered":"2024-03-28T15:10:38.930520+0000","last_clean":"2024-03-28T15:10:38.930520+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:10:38.930520+0000","last_undegraded":"2024-03-28T15:10:38.930520+0000","last_fullsized":"2024-03-28T15:10:38.930520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:43.533807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019334860000000001,"stat_sum":{"num_bytes":1345083,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":314,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"37'329","reported_seq":3088,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.922289+0000","last_change":"2024-03-28T15:04:56.198079+0000","last_active":"2024-03-28T15:10:38.922289+0000","last_peered":"2024-03-28T15:10:38.922289+0000","last_clean":"2024-03-28T15:10:38.922289+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:10:38.922289+0000","last_undegraded":"2024-03-28T15:10:38.922289+0000","last_fullsized":"2024-03-28T15:10:38.922289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:00:42.406838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025151280000000002,"stat_sum":{"num_bytes":12619399,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":685,"num_read_kb":17683,"num_write":321,"num_write_kb":20675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"37'314","reported_seq":3077,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938235+0000","last_change":"2024-03-28T15:04:56.747212+0000","last_active":"2024-03-28T15:10:38.938235+0000","last_peered":"2024-03-28T15:10:38.938235+0000","last_clean":"2024-03-28T15:10:38.938235+0000","last_became_active":"2024-03-28T15:02:12.485356+0000","last_became_peered":"2024-03-28T15:02:12.485356+0000","last_unstale":"2024-03-28T15:10:38.938235+0000","last_undegraded":"2024-03-28T15:10:38.938235+0000","last_fullsized":"2024-03-28T15:10:38.938235+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:51:41.708849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014203849999999999,"stat_sum":{"num_bytes":19355212,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":695,"num_read_kb":14278,"num_write":306,"num_write_kb":19012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"37'363","reported_seq":2551,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938805+0000","last_change":"2024-03-28T15:04:56.745133+0000","last_active":"2024-03-28T15:10:38.938805+0000","last_peered":"2024-03-28T15:10:38.938805+0000","last_clean":"2024-03-28T15:10:38.938805+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:10:38.938805+0000","last_undegraded":"2024-03-28T15:10:38.938805+0000","last_fullsized":"2024-03-28T15:10:38.938805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:21.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002796064,"stat_sum":{"num_bytes":10679980,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":115,"num_read_kb":4114,"num_write":361,"num_write_kb":10554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"37'327","reported_seq":2639,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.946373+0000","last_change":"2024-03-28T15:04:56.200817+0000","last_active":"2024-03-28T15:10:38.946373+0000","last_peered":"2024-03-28T15:10:38.946373+0000","last_clean":"2024-03-28T15:10:38.946373+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:10:38.946373+0000","last_undegraded":"2024-03-28T15:10:38.946373+0000","last_fullsized":"2024-03-28T15:10:38.946373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:03:09.771351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003798718,"stat_sum":{"num_bytes":10128543,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":239,"num_read_kb":4123,"num_write":321,"num_write_kb":10056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"37'354","reported_seq":3455,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.950756+0000","last_change":"2024-03-28T15:04:56.743147+0000","last_active":"2024-03-28T15:10:38.950756+0000","last_peered":"2024-03-28T15:10:38.950756+0000","last_clean":"2024-03-28T15:10:38.950756+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:10:38.950756+0000","last_undegraded":"2024-03-28T15:10:38.950756+0000","last_fullsized":"2024-03-28T15:10:38.950756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:20.151357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001484766,"stat_sum":{"num_bytes":23645872,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":1028,"num_read_kb":17521,"num_write":350,"num_write_kb":23210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"37'334","reported_seq":2944,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.959962+0000","last_change":"2024-03-28T15:04:56.208811+0000","last_active":"2024-03-28T15:10:38.959962+0000","last_peered":"2024-03-28T15:10:38.959962+0000","last_clean":"2024-03-28T15:10:38.959962+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:10:38.959962+0000","last_undegraded":"2024-03-28T15:10:38.959962+0000","last_fullsized":"2024-03-28T15:10:38.959962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:57.578217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017148910000000001,"stat_sum":{"num_bytes":19312172,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":537,"num_read_kb":7179,"num_write":326,"num_write_kb":18943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"37'330","reported_seq":2712,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960654+0000","last_change":"2024-03-28T15:04:56.745714+0000","last_active":"2024-03-28T15:10:38.960654+0000","last_peered":"2024-03-28T15:10:38.960654+0000","last_clean":"2024-03-28T15:10:38.960654+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:10:38.960654+0000","last_undegraded":"2024-03-28T15:10:38.960654+0000","last_fullsized":"2024-03-28T15:10:38.960654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:27:32.222447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023275240000000001,"stat_sum":{"num_bytes":6398085,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":309,"num_read_kb":8863,"num_write":320,"num_write_kb":10489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"37'327","reported_seq":3210,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.961682+0000","last_change":"2024-03-28T15:04:56.534451+0000","last_active":"2024-03-28T15:10:38.961682+0000","last_peered":"2024-03-28T15:10:38.961682+0000","last_clean":"2024-03-28T15:10:38.961682+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:10:38.961682+0000","last_undegraded":"2024-03-28T15:10:38.961682+0000","last_fullsized":"2024-03-28T15:10:38.961682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:43.533807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019348060000000001,"stat_sum":{"num_bytes":19157707,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":810,"num_read_kb":17508,"num_write":311,"num_write_kb":18862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"37'303","reported_seq":3011,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.959082+0000","last_change":"2024-03-28T15:04:56.534170+0000","last_active":"2024-03-28T15:10:38.959082+0000","last_peered":"2024-03-28T15:10:38.959082+0000","last_clean":"2024-03-28T15:10:38.959082+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:10:38.959082+0000","last_undegraded":"2024-03-28T15:10:38.959082+0000","last_fullsized":"2024-03-28T15:10:38.959082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:20.495757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018723170000000001,"stat_sum":{"num_bytes":18249049,"num_objects":129,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":635,"num_read_kb":13018,"num_write":297,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"37'296","reported_seq":2603,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.934653+0000","last_change":"2024-03-28T15:04:56.199593+0000","last_active":"2024-03-28T15:10:38.934653+0000","last_peered":"2024-03-28T15:10:38.934653+0000","last_clean":"2024-03-28T15:10:38.934653+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:10:38.934653+0000","last_undegraded":"2024-03-28T15:10:38.934653+0000","last_fullsized":"2024-03-28T15:10:38.934653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:50.527037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013925400000000001,"stat_sum":{"num_bytes":6129603,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":234,"num_read_kb":4757,"num_write":288,"num_write_kb":10236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"37'297","reported_seq":2859,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936011+0000","last_change":"2024-03-28T15:04:56.207750+0000","last_active":"2024-03-28T15:10:38.936011+0000","last_peered":"2024-03-28T15:10:38.936011+0000","last_clean":"2024-03-28T15:10:38.936011+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:10:38.936011+0000","last_undegraded":"2024-03-28T15:10:38.936011+0000","last_fullsized":"2024-03-28T15:10:38.936011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:17:15.635128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017572180000000001,"stat_sum":{"num_bytes":15029757,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":489,"num_read_kb":12548,"num_write":286,"num_write_kb":23016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"37'341","reported_seq":2613,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932033+0000","last_change":"2024-03-28T15:04:56.197644+0000","last_active":"2024-03-28T15:10:38.932033+0000","last_peered":"2024-03-28T15:10:38.932033+0000","last_clean":"2024-03-28T15:10:38.932033+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:10:38.932033+0000","last_undegraded":"2024-03-28T15:10:38.932033+0000","last_fullsized":"2024-03-28T15:10:38.932033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:08.587017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002253971,"stat_sum":{"num_bytes":6664925,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":199,"num_read_kb":4930,"num_write":335,"num_write_kb":6638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"37'357","reported_seq":3528,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.926788+0000","last_change":"2024-03-28T15:04:56.293889+0000","last_active":"2024-03-28T15:10:38.926788+0000","last_peered":"2024-03-28T15:10:38.926788+0000","last_clean":"2024-03-28T15:10:38.926788+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:10:38.926788+0000","last_undegraded":"2024-03-28T15:10:38.926788+0000","last_fullsized":"2024-03-28T15:10:38.926788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:20.495757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179673,"stat_sum":{"num_bytes":23044368,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":1098,"num_read_kb":19203,"num_write":350,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"37'357","reported_seq":2912,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.945394+0000","last_change":"2024-03-28T15:04:56.200579+0000","last_active":"2024-03-28T15:10:38.945394+0000","last_peered":"2024-03-28T15:10:38.945394+0000","last_clean":"2024-03-28T15:10:38.945394+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:10:38.945394+0000","last_undegraded":"2024-03-28T15:10:38.945394+0000","last_fullsized":"2024-03-28T15:10:38.945394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:30.228263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039634529999999996,"stat_sum":{"num_bytes":16147473,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":482,"num_read_kb":8871,"num_write":348,"num_write_kb":19203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"37'349","reported_seq":2573,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960198+0000","last_change":"2024-03-28T15:04:56.201087+0000","last_active":"2024-03-28T15:10:38.960198+0000","last_peered":"2024-03-28T15:10:38.960198+0000","last_clean":"2024-03-28T15:10:38.960198+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:10:38.960198+0000","last_undegraded":"2024-03-28T15:10:38.960198+0000","last_fullsized":"2024-03-28T15:10:38.960198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:36.687483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027424319999999999,"stat_sum":{"num_bytes":10498072,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":151,"num_read_kb":9005,"num_write":340,"num_write_kb":14498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"37'311","reported_seq":2414,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936309+0000","last_change":"2024-03-28T15:04:56.746916+0000","last_active":"2024-03-28T15:10:38.936309+0000","last_peered":"2024-03-28T15:10:38.936309+0000","last_clean":"2024-03-28T15:10:38.936309+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:10:38.936309+0000","last_undegraded":"2024-03-28T15:10:38.936309+0000","last_fullsized":"2024-03-28T15:10:38.936309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:20:35.393196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179399,"stat_sum":{"num_bytes":9508598,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":30,"num_read_kb":42,"num_write":300,"num_write_kb":9445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"37'324","reported_seq":2710,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939190+0000","last_change":"2024-03-28T15:04:56.745421+0000","last_active":"2024-03-28T15:10:38.939190+0000","last_peered":"2024-03-28T15:10:38.939190+0000","last_clean":"2024-03-28T15:10:38.939190+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:10:38.939190+0000","last_undegraded":"2024-03-28T15:10:38.939190+0000","last_fullsized":"2024-03-28T15:10:38.939190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:50.382005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030062439999999999,"stat_sum":{"num_bytes":14185939,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":312,"num_read_kb":12813,"num_write":317,"num_write_kb":18119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"37'356","reported_seq":3288,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937246+0000","last_change":"2024-03-28T15:04:56.534664+0000","last_active":"2024-03-28T15:10:38.937246+0000","last_peered":"2024-03-28T15:10:38.937246+0000","last_clean":"2024-03-28T15:10:38.937246+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:10:38.937246+0000","last_undegraded":"2024-03-28T15:10:38.937246+0000","last_fullsized":"2024-03-28T15:10:38.937246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:10.614269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001798584,"stat_sum":{"num_bytes":30036364,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":858,"num_read_kb":15492,"num_write":348,"num_write_kb":37706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"37'342","reported_seq":2724,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957518+0000","last_change":"2024-03-28T15:04:56.198376+0000","last_active":"2024-03-28T15:10:38.957518+0000","last_peered":"2024-03-28T15:10:38.957518+0000","last_clean":"2024-03-28T15:10:38.957518+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:10:38.957518+0000","last_undegraded":"2024-03-28T15:10:38.957518+0000","last_fullsized":"2024-03-28T15:10:38.957518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:55:39.836954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025216930000000002,"stat_sum":{"num_bytes":10421413,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":308,"num_read_kb":8356,"num_write":331,"num_write_kb":10350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"37'337","reported_seq":2523,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935072+0000","last_change":"2024-03-28T15:04:56.746844+0000","last_active":"2024-03-28T15:10:38.935072+0000","last_peered":"2024-03-28T15:10:38.935072+0000","last_clean":"2024-03-28T15:10:38.935072+0000","last_became_active":"2024-03-28T15:02:12.178196+0000","last_became_peered":"2024-03-28T15:02:12.178196+0000","last_unstale":"2024-03-28T15:10:38.935072+0000","last_undegraded":"2024-03-28T15:10:38.935072+0000","last_fullsized":"2024-03-28T15:10:38.935072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:59:11.409442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001195878,"stat_sum":{"num_bytes":9431253,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":119,"num_read_kb":4113,"num_write":327,"num_write_kb":13479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"37'331","reported_seq":2998,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939818+0000","last_change":"2024-03-28T15:04:56.207990+0000","last_active":"2024-03-28T15:10:38.939818+0000","last_peered":"2024-03-28T15:10:38.939818+0000","last_clean":"2024-03-28T15:10:38.939818+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:10:38.939818+0000","last_undegraded":"2024-03-28T15:10:38.939818+0000","last_fullsized":"2024-03-28T15:10:38.939818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:51:56.779182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020004409999999999,"stat_sum":{"num_bytes":16319296,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":593,"num_read_kb":18222,"num_write":323,"num_write_kb":20203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"37'356","reported_seq":2799,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956957+0000","last_change":"2024-03-28T15:04:56.200922+0000","last_active":"2024-03-28T15:10:38.956957+0000","last_peered":"2024-03-28T15:10:38.956957+0000","last_clean":"2024-03-28T15:10:38.956957+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:10:38.956957+0000","last_undegraded":"2024-03-28T15:10:38.956957+0000","last_fullsized":"2024-03-28T15:10:38.956957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:57.578217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031981639999999999,"stat_sum":{"num_bytes":14893759,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":369,"num_read_kb":10097,"num_write":349,"num_write_kb":14727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"37'314","reported_seq":3255,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.931560+0000","last_change":"2024-03-28T15:04:56.207665+0000","last_active":"2024-03-28T15:10:38.931560+0000","last_peered":"2024-03-28T15:10:38.931560+0000","last_clean":"2024-03-28T15:10:38.931560+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:10:38.931560+0000","last_undegraded":"2024-03-28T15:10:38.931560+0000","last_fullsized":"2024-03-28T15:10:38.931560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:36:17.765596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001813119,"stat_sum":{"num_bytes":18699537,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":867,"num_read_kb":15099,"num_write":310,"num_write_kb":18421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"37'377","reported_seq":2618,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960107+0000","last_change":"2024-03-28T15:04:56.534606+0000","last_active":"2024-03-28T15:10:38.960107+0000","last_peered":"2024-03-28T15:10:38.960107+0000","last_clean":"2024-03-28T15:10:38.960107+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:10:38.960107+0000","last_undegraded":"2024-03-28T15:10:38.960107+0000","last_fullsized":"2024-03-28T15:10:38.960107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:23:24.123040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019272180000000001,"stat_sum":{"num_bytes":11206818,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":167,"num_read_kb":4115,"num_write":363,"num_write_kb":11124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"37'320","reported_seq":2923,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936502+0000","last_change":"2024-03-28T15:04:56.294225+0000","last_active":"2024-03-28T15:10:38.936502+0000","last_peered":"2024-03-28T15:10:38.936502+0000","last_clean":"2024-03-28T15:10:38.936502+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:10:38.936502+0000","last_undegraded":"2024-03-28T15:10:38.936502+0000","last_fullsized":"2024-03-28T15:10:38.936502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:23:24.123040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019373070000000001,"stat_sum":{"num_bytes":22270707,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":529,"num_read_kb":12598,"num_write":314,"num_write_kb":21911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"37'315","reported_seq":3227,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.942957+0000","last_change":"2024-03-28T15:04:56.208607+0000","last_active":"2024-03-28T15:10:38.942957+0000","last_peered":"2024-03-28T15:10:38.942957+0000","last_clean":"2024-03-28T15:10:38.942957+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:10:38.942957+0000","last_undegraded":"2024-03-28T15:10:38.942957+0000","last_fullsized":"2024-03-28T15:10:38.942957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:03:09.771351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020980170000000002,"stat_sum":{"num_bytes":17909578,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":838,"num_read_kb":18539,"num_write":311,"num_write_kb":20928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"37'358","reported_seq":3192,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935632+0000","last_change":"2024-03-28T15:04:56.198529+0000","last_active":"2024-03-28T15:10:38.935632+0000","last_peered":"2024-03-28T15:10:38.935632+0000","last_clean":"2024-03-28T15:10:38.935632+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:10:38.935632+0000","last_undegraded":"2024-03-28T15:10:38.935632+0000","last_fullsized":"2024-03-28T15:10:38.935632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:41:01.927661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026139499999999999,"stat_sum":{"num_bytes":26288347,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":760,"num_read_kb":8154,"num_write":350,"num_write_kb":25854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"37'359","reported_seq":2900,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956962+0000","last_change":"2024-03-28T15:04:56.200966+0000","last_active":"2024-03-28T15:10:38.956962+0000","last_peered":"2024-03-28T15:10:38.956962+0000","last_clean":"2024-03-28T15:10:38.956962+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:10:38.956962+0000","last_undegraded":"2024-03-28T15:10:38.956962+0000","last_fullsized":"2024-03-28T15:10:38.956962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:37.204800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036450829999999999,"stat_sum":{"num_bytes":10387544,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":467,"num_read_kb":8853,"num_write":352,"num_write_kb":10266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"37'386","reported_seq":2968,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.942998+0000","last_change":"2024-03-28T15:04:56.747393+0000","last_active":"2024-03-28T15:10:38.942998+0000","last_peered":"2024-03-28T15:10:38.942998+0000","last_clean":"2024-03-28T15:10:38.942998+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:10:38.942998+0000","last_undegraded":"2024-03-28T15:10:38.942998+0000","last_fullsized":"2024-03-28T15:10:38.942998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:39:11.217145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001020962,"stat_sum":{"num_bytes":12939697,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":508,"num_read_kb":8898,"num_write":372,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"37'384","reported_seq":2787,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932311+0000","last_change":"2024-03-28T15:04:56.534204+0000","last_active":"2024-03-28T15:10:38.932311+0000","last_peered":"2024-03-28T15:10:38.932311+0000","last_clean":"2024-03-28T15:10:38.932311+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:10:38.932311+0000","last_undegraded":"2024-03-28T15:10:38.932311+0000","last_fullsized":"2024-03-28T15:10:38.932311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:56:09.141610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019055229999999999,"stat_sum":{"num_bytes":11304826,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":329,"num_read_kb":5287,"num_write":374,"num_write_kb":11236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"37'322","reported_seq":2854,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.921473+0000","last_change":"2024-03-28T15:04:56.196629+0000","last_active":"2024-03-28T15:10:38.921473+0000","last_peered":"2024-03-28T15:10:38.921473+0000","last_clean":"2024-03-28T15:10:38.921473+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:10:38.921473+0000","last_undegraded":"2024-03-28T15:10:38.921473+0000","last_fullsized":"2024-03-28T15:10:38.921473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:51.357070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078031800000000001,"stat_sum":{"num_bytes":14482293,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":459,"num_read_kb":10396,"num_write":315,"num_write_kb":14305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"36'18","reported_seq":74,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.776351+0000","last_change":"2024-03-28T15:01:31.097415+0000","last_active":"2024-03-28T15:09:56.776351+0000","last_peered":"2024-03-28T15:09:56.776351+0000","last_clean":"2024-03-28T15:09:56.776351+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:09:56.776351+0000","last_undegraded":"2024-03-28T15:09:56.776351+0000","last_fullsized":"2024-03-28T15:09:56.776351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":16384,"num_write":21,"num_write_kb":4230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"37'325","reported_seq":2638,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.934802+0000","last_change":"2024-03-28T15:04:56.198560+0000","last_active":"2024-03-28T15:10:38.934802+0000","last_peered":"2024-03-28T15:10:38.934802+0000","last_clean":"2024-03-28T15:10:38.934802+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:10:38.934802+0000","last_undegraded":"2024-03-28T15:10:38.934802+0000","last_fullsized":"2024-03-28T15:10:38.934802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:51.739886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023690579999999998,"stat_sum":{"num_bytes":11043752,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":240,"num_read_kb":4317,"num_write":317,"num_write_kb":10950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"37'647","reported_seq":711,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.472792+0000","last_change":"2024-03-28T15:01:31.088437+0000","last_active":"2024-03-28T15:10:02.472792+0000","last_peered":"2024-03-28T15:10:02.472792+0000","last_clean":"2024-03-28T15:10:02.472792+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:10:02.472792+0000","last_undegraded":"2024-03-28T15:10:02.472792+0000","last_fullsized":"2024-03-28T15:10:02.472792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":34526,"num_write":642,"num_write_kb":12757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"36'58","reported_seq":112,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.775891+0000","last_change":"2024-03-28T15:01:31.097499+0000","last_active":"2024-03-28T15:09:56.775891+0000","last_peered":"2024-03-28T15:09:56.775891+0000","last_clean":"2024-03-28T15:09:56.775891+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:09:56.775891+0000","last_undegraded":"2024-03-28T15:09:56.775891+0000","last_fullsized":"2024-03-28T15:09:56.775891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":62,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"37'402","reported_seq":3693,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936725+0000","last_change":"2024-03-28T15:04:56.208655+0000","last_active":"2024-03-28T15:10:38.936725+0000","last_peered":"2024-03-28T15:10:38.936725+0000","last_clean":"2024-03-28T15:10:38.936725+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:10:38.936725+0000","last_undegraded":"2024-03-28T15:10:38.936725+0000","last_fullsized":"2024-03-28T15:10:38.936725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:37.204800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017492600000000001,"stat_sum":{"num_bytes":22338843,"num_objects":188,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":1218,"num_read_kb":22454,"num_write":394,"num_write_kb":30214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"37'604","reported_seq":661,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:30.364435+0000","last_change":"2024-03-28T15:01:31.086252+0000","last_active":"2024-03-28T15:10:30.364435+0000","last_peered":"2024-03-28T15:10:30.364435+0000","last_clean":"2024-03-28T15:10:30.364435+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:10:30.364435+0000","last_undegraded":"2024-03-28T15:10:30.364435+0000","last_fullsized":"2024-03-28T15:10:30.364435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":603,"num_write_kb":12492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"37'341","reported_seq":3347,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937426+0000","last_change":"2024-03-28T15:04:56.293936+0000","last_active":"2024-03-28T15:10:38.937426+0000","last_peered":"2024-03-28T15:10:38.937426+0000","last_clean":"2024-03-28T15:10:38.937426+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:10:38.937426+0000","last_undegraded":"2024-03-28T15:10:38.937426+0000","last_fullsized":"2024-03-28T15:10:38.937426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:03.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022536280000000001,"stat_sum":{"num_bytes":19647424,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":933,"num_read_kb":22521,"num_write":330,"num_write_kb":23406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"37'5","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.246758+0000","last_change":"2024-03-28T15:01:31.092338+0000","last_active":"2024-03-28T15:09:56.246758+0000","last_peered":"2024-03-28T15:09:56.246758+0000","last_clean":"2024-03-28T15:09:56.246758+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:09:56.246758+0000","last_undegraded":"2024-03-28T15:09:56.246758+0000","last_fullsized":"2024-03-28T15:09:56.246758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"37'366","reported_seq":3228,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960208+0000","last_change":"2024-03-28T15:04:56.200271+0000","last_active":"2024-03-28T15:10:38.960208+0000","last_peered":"2024-03-28T15:10:38.960208+0000","last_clean":"2024-03-28T15:10:38.960208+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:10:38.960208+0000","last_undegraded":"2024-03-28T15:10:38.960208+0000","last_fullsized":"2024-03-28T15:10:38.960208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:36:17.765596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043389099999999996,"stat_sum":{"num_bytes":22872414,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":789,"num_read_kb":19564,"num_write":357,"num_write_kb":22493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"37'70","reported_seq":124,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.349608+0000","last_change":"2024-03-28T15:01:31.088909+0000","last_active":"2024-03-28T15:10:02.349608+0000","last_peered":"2024-03-28T15:10:02.349608+0000","last_clean":"2024-03-28T15:10:02.349608+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:10:02.349608+0000","last_undegraded":"2024-03-28T15:10:02.349608+0000","last_fullsized":"2024-03-28T15:10:02.349608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":66,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"37'323","reported_seq":2969,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.943307+0000","last_change":"2024-03-28T15:04:56.197873+0000","last_active":"2024-03-28T15:10:38.943307+0000","last_peered":"2024-03-28T15:10:38.943307+0000","last_clean":"2024-03-28T15:10:38.943307+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:10:38.943307+0000","last_undegraded":"2024-03-28T15:10:38.943307+0000","last_fullsized":"2024-03-28T15:10:38.943307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:52.072532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023945189999999999,"stat_sum":{"num_bytes":10186807,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":573,"num_read_kb":8919,"num_write":314,"num_write_kb":10118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"36'193","reported_seq":257,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.698311+0000","last_change":"2024-03-28T15:01:31.095935+0000","last_active":"2024-03-28T15:10:02.698311+0000","last_peered":"2024-03-28T15:10:02.698311+0000","last_clean":"2024-03-28T15:10:02.698311+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:10:02.698311+0000","last_undegraded":"2024-03-28T15:10:02.698311+0000","last_fullsized":"2024-03-28T15:10:02.698311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":4122,"num_write":194,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"37'321","reported_seq":3146,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957720+0000","last_change":"2024-03-28T15:04:56.199307+0000","last_active":"2024-03-28T15:10:38.957720+0000","last_peered":"2024-03-28T15:10:38.957720+0000","last_clean":"2024-03-28T15:10:38.957720+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:10:38.957720+0000","last_undegraded":"2024-03-28T15:10:38.957720+0000","last_fullsized":"2024-03-28T15:10:38.957720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:32:03.268809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254104,"stat_sum":{"num_bytes":14752712,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":752,"num_read_kb":16498,"num_write":310,"num_write_kb":18664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"37'122","reported_seq":201,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.389422+0000","last_change":"2024-03-28T15:01:31.089380+0000","last_active":"2024-03-28T15:10:29.389422+0000","last_peered":"2024-03-28T15:10:29.389422+0000","last_clean":"2024-03-28T15:10:29.389422+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:10:29.389422+0000","last_undegraded":"2024-03-28T15:10:29.389422+0000","last_fullsized":"2024-03-28T15:10:29.389422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32804,"num_write":115,"num_write_kb":12363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"37'335","reported_seq":2431,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939280+0000","last_change":"2024-03-28T15:04:56.745455+0000","last_active":"2024-03-28T15:10:38.939280+0000","last_peered":"2024-03-28T15:10:38.939280+0000","last_clean":"2024-03-28T15:10:38.939280+0000","last_became_active":"2024-03-28T15:02:12.182402+0000","last_became_peered":"2024-03-28T15:02:12.182402+0000","last_unstale":"2024-03-28T15:10:38.939280+0000","last_undegraded":"2024-03-28T15:10:38.939280+0000","last_fullsized":"2024-03-28T15:10:38.939280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:32:46.642308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030686839999999999,"stat_sum":{"num_bytes":2787093,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":329,"num_write_kb":2901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"36'96","reported_seq":163,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.488638+0000","last_change":"2024-03-28T15:01:31.098172+0000","last_active":"2024-03-28T15:10:01.488638+0000","last_peered":"2024-03-28T15:10:01.488638+0000","last_clean":"2024-03-28T15:10:01.488638+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:10:01.488638+0000","last_undegraded":"2024-03-28T15:10:01.488638+0000","last_fullsized":"2024-03-28T15:10:01.488638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:16.015528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24616,"num_write":94,"num_write_kb":8237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"37'348","reported_seq":3089,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.928031+0000","last_change":"2024-03-28T15:04:56.294429+0000","last_active":"2024-03-28T15:10:38.928031+0000","last_peered":"2024-03-28T15:10:38.928031+0000","last_clean":"2024-03-28T15:10:38.928031+0000","last_became_active":"2024-03-28T15:02:12.180011+0000","last_became_peered":"2024-03-28T15:02:12.180011+0000","last_unstale":"2024-03-28T15:10:38.928031+0000","last_undegraded":"2024-03-28T15:10:38.928031+0000","last_fullsized":"2024-03-28T15:10:38.928031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:43.521019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018715680000000001,"stat_sum":{"num_bytes":20266861,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":674,"num_read_kb":16415,"num_write":337,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":51,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.247634+0000","last_change":"2024-03-28T15:01:31.090519+0000","last_active":"2024-03-28T15:09:56.247634+0000","last_peered":"2024-03-28T15:09:56.247634+0000","last_clean":"2024-03-28T15:09:56.247634+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:09:56.247634+0000","last_undegraded":"2024-03-28T15:09:56.247634+0000","last_fullsized":"2024-03-28T15:09:56.247634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"37'358","reported_seq":2567,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.952909+0000","last_change":"2024-03-28T15:04:56.744254+0000","last_active":"2024-03-28T15:10:38.952909+0000","last_peered":"2024-03-28T15:10:38.952909+0000","last_clean":"2024-03-28T15:10:38.952909+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:10:38.952909+0000","last_undegraded":"2024-03-28T15:10:38.952909+0000","last_fullsized":"2024-03-28T15:10:38.952909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:37.876534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022269199999999999,"stat_sum":{"num_bytes":11015620,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":136,"num_read_kb":3039,"num_write":343,"num_write_kb":10934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"37'4","reported_seq":55,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:10.282196+0000","last_change":"2024-03-28T15:01:31.088364+0000","last_active":"2024-03-28T15:10:10.282196+0000","last_peered":"2024-03-28T15:10:10.282196+0000","last_clean":"2024-03-28T15:10:10.282196+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:10:10.282196+0000","last_undegraded":"2024-03-28T15:10:10.282196+0000","last_fullsized":"2024-03-28T15:10:10.282196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"37'344","reported_seq":2987,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.931143+0000","last_change":"2024-03-28T15:04:56.534032+0000","last_active":"2024-03-28T15:10:38.931143+0000","last_peered":"2024-03-28T15:10:38.931143+0000","last_clean":"2024-03-28T15:10:38.931143+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:10:38.931143+0000","last_undegraded":"2024-03-28T15:10:38.931143+0000","last_fullsized":"2024-03-28T15:10:38.931143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:03.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001897143,"stat_sum":{"num_bytes":17657480,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":571,"num_read_kb":16762,"num_write":332,"num_write_kb":17420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"36'220","reported_seq":300,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:23.112150+0000","last_change":"2024-03-28T15:01:31.094837+0000","last_active":"2024-03-28T15:10:23.112150+0000","last_peered":"2024-03-28T15:10:23.112150+0000","last_clean":"2024-03-28T15:10:23.112150+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:10:23.112150+0000","last_undegraded":"2024-03-28T15:10:23.112150+0000","last_fullsized":"2024-03-28T15:10:23.112150+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":83,"num_read_kb":8273,"num_write":219,"num_write_kb":8316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"37'366","reported_seq":2554,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.953014+0000","last_change":"2024-03-28T15:04:56.743437+0000","last_active":"2024-03-28T15:10:38.953014+0000","last_peered":"2024-03-28T15:10:38.953014+0000","last_clean":"2024-03-28T15:10:38.953014+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:10:38.953014+0000","last_undegraded":"2024-03-28T15:10:38.953014+0000","last_fullsized":"2024-03-28T15:10:38.953014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:12:47.009853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001763492,"stat_sum":{"num_bytes":16350020,"num_objects":186,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":115,"num_read_kb":12505,"num_write":360,"num_write_kb":24351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"37'28","reported_seq":88,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:32.560745+0000","last_change":"2024-03-28T15:01:31.092455+0000","last_active":"2024-03-28T15:10:32.560745+0000","last_peered":"2024-03-28T15:10:32.560745+0000","last_clean":"2024-03-28T15:10:32.560745+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:10:32.560745+0000","last_undegraded":"2024-03-28T15:10:32.560745+0000","last_fullsized":"2024-03-28T15:10:32.560745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:04.636216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":29,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"37'290","reported_seq":2751,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.954983+0000","last_change":"2024-03-28T15:04:56.745120+0000","last_active":"2024-03-28T15:10:38.954983+0000","last_peered":"2024-03-28T15:10:38.954983+0000","last_clean":"2024-03-28T15:10:38.954983+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:10:38.954983+0000","last_undegraded":"2024-03-28T15:10:38.954983+0000","last_fullsized":"2024-03-28T15:10:38.954983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:28:19.525504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002782082,"stat_sum":{"num_bytes":11071386,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":387,"num_read_kb":12458,"num_write":285,"num_write_kb":15052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"26'28","reported_seq":90,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:24.170170+0000","last_change":"2024-03-28T15:01:31.092175+0000","last_active":"2024-03-28T15:10:24.170170+0000","last_peered":"2024-03-28T15:10:24.170170+0000","last_clean":"2024-03-28T15:10:24.170170+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:10:24.170170+0000","last_undegraded":"2024-03-28T15:10:24.170170+0000","last_fullsized":"2024-03-28T15:10:24.170170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":28,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"37'337","reported_seq":2700,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957003+0000","last_change":"2024-03-28T15:04:56.745998+0000","last_active":"2024-03-28T15:10:38.957003+0000","last_peered":"2024-03-28T15:10:38.957003+0000","last_clean":"2024-03-28T15:10:38.957003+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:10:38.957003+0000","last_undegraded":"2024-03-28T15:10:38.957003+0000","last_fullsized":"2024-03-28T15:10:38.957003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:36:25.325471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025006389999999998,"stat_sum":{"num_bytes":10266965,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":289,"num_read_kb":13000,"num_write":328,"num_write_kb":14296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"37'120","reported_seq":180,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:34.394134+0000","last_change":"2024-03-28T15:01:31.097901+0000","last_active":"2024-03-28T15:10:34.394134+0000","last_peered":"2024-03-28T15:10:34.394134+0000","last_clean":"2024-03-28T15:10:34.394134+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:10:34.394134+0000","last_undegraded":"2024-03-28T15:10:34.394134+0000","last_fullsized":"2024-03-28T15:10:34.394134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":125,"num_write_kb":4282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"37'362","reported_seq":2665,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957255+0000","last_change":"2024-03-28T15:04:56.200770+0000","last_active":"2024-03-28T15:10:38.957255+0000","last_peered":"2024-03-28T15:10:38.957255+0000","last_clean":"2024-03-28T15:10:38.957255+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:10:38.957255+0000","last_undegraded":"2024-03-28T15:10:38.957255+0000","last_fullsized":"2024-03-28T15:10:38.957255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:09.229020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003899156,"stat_sum":{"num_bytes":10797940,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":229,"num_read_kb":5058,"num_write":352,"num_write_kb":10729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"37'10","reported_seq":62,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:34.525786+0000","last_change":"2024-03-28T15:01:31.086634+0000","last_active":"2024-03-28T15:10:34.525786+0000","last_peered":"2024-03-28T15:10:34.525786+0000","last_clean":"2024-03-28T15:10:34.525786+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:10:34.525786+0000","last_undegraded":"2024-03-28T15:10:34.525786+0000","last_fullsized":"2024-03-28T15:10:34.525786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":4136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"37'326","reported_seq":2660,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957715+0000","last_change":"2024-03-28T15:04:56.744447+0000","last_active":"2024-03-28T15:10:38.957715+0000","last_peered":"2024-03-28T15:10:38.957715+0000","last_clean":"2024-03-28T15:10:38.957715+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:10:38.957715+0000","last_undegraded":"2024-03-28T15:10:38.957715+0000","last_fullsized":"2024-03-28T15:10:38.957715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:43.786965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026010120000000002,"stat_sum":{"num_bytes":14545241,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":260,"num_read_kb":2397,"num_write":318,"num_write_kb":18466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":208,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:57.047088+0000","last_change":"2024-03-28T15:01:20.724163+0000","last_active":"2024-03-28T15:09:57.047088+0000","last_peered":"2024-03-28T15:09:57.047088+0000","last_clean":"2024-03-28T15:09:57.047088+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:09:57.047088+0000","last_undegraded":"2024-03-28T15:09:57.047088+0000","last_fullsized":"2024-03-28T15:09:57.047088+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_clean_scrub_stamp":"2024-03-28T15:01:19.548090+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:15.791893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"37'84","reported_seq":137,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:03.245313+0000","last_change":"2024-03-28T15:01:31.085453+0000","last_active":"2024-03-28T15:10:03.245313+0000","last_peered":"2024-03-28T15:10:03.245313+0000","last_clean":"2024-03-28T15:10:03.245313+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:10:03.245313+0000","last_undegraded":"2024-03-28T15:10:03.245313+0000","last_fullsized":"2024-03-28T15:10:03.245313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":85,"num_write_kb":4165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"37'326","reported_seq":2489,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960193+0000","last_change":"2024-03-28T15:04:56.208514+0000","last_active":"2024-03-28T15:10:38.960193+0000","last_peered":"2024-03-28T15:10:38.960193+0000","last_clean":"2024-03-28T15:10:38.960193+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:10:38.960193+0000","last_undegraded":"2024-03-28T15:10:38.960193+0000","last_fullsized":"2024-03-28T15:10:38.960193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:09.229020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002201228,"stat_sum":{"num_bytes":6136325,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":89,"num_read_kb":4109,"num_write":317,"num_write_kb":10222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"37'225","reported_seq":285,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:30.368284+0000","last_change":"2024-03-28T15:01:31.095439+0000","last_active":"2024-03-28T15:10:30.368284+0000","last_peered":"2024-03-28T15:10:30.368284+0000","last_clean":"2024-03-28T15:10:30.368284+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:10:30.368284+0000","last_undegraded":"2024-03-28T15:10:30.368284+0000","last_fullsized":"2024-03-28T15:10:30.368284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":10,"num_read_kb":28674,"num_write":226,"num_write_kb":12578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"37'330","reported_seq":2633,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932587+0000","last_change":"2024-03-28T15:04:56.198312+0000","last_active":"2024-03-28T15:10:38.932587+0000","last_peered":"2024-03-28T15:10:38.932587+0000","last_clean":"2024-03-28T15:10:38.932587+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:10:38.932587+0000","last_undegraded":"2024-03-28T15:10:38.932587+0000","last_fullsized":"2024-03-28T15:10:38.932587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:30.445328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026072840000000001,"stat_sum":{"num_bytes":10172814,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":229,"num_read_kb":5026,"num_write":323,"num_write_kb":10109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'80","reported_seq":209,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:10.281992+0000","last_change":"2024-03-28T15:01:31.093265+0000","last_active":"2024-03-28T15:10:10.281992+0000","last_peered":"2024-03-28T15:10:10.281992+0000","last_clean":"2024-03-28T15:10:10.281992+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:10:10.281992+0000","last_undegraded":"2024-03-28T15:10:10.281992+0000","last_fullsized":"2024-03-28T15:10:10.281992+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534189,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":569,"num_write":78,"num_write_kb":592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"37'335","reported_seq":2426,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937905+0000","last_change":"2024-03-28T15:04:56.293657+0000","last_active":"2024-03-28T15:10:38.937905+0000","last_peered":"2024-03-28T15:10:38.937905+0000","last_clean":"2024-03-28T15:10:38.937905+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:10:38.937905+0000","last_undegraded":"2024-03-28T15:10:38.937905+0000","last_fullsized":"2024-03-28T15:10:38.937905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:56:09.141610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017724730000000001,"stat_sum":{"num_bytes":1382309,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":326,"num_write_kb":1524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"25'82","reported_seq":140,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.264455+0000","last_change":"2024-03-28T15:01:31.094750+0000","last_active":"2024-03-28T15:10:01.264455+0000","last_peered":"2024-03-28T15:10:01.264455+0000","last_clean":"2024-03-28T15:10:01.264455+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:10:01.264455+0000","last_undegraded":"2024-03-28T15:10:01.264455+0000","last_fullsized":"2024-03-28T15:10:01.264455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":80,"num_write_kb":8230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"37'356","reported_seq":2673,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.933205+0000","last_change":"2024-03-28T15:04:56.198147+0000","last_active":"2024-03-28T15:10:38.933205+0000","last_peered":"2024-03-28T15:10:38.933205+0000","last_clean":"2024-03-28T15:10:38.933205+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:10:38.933205+0000","last_undegraded":"2024-03-28T15:10:38.933205+0000","last_fullsized":"2024-03-28T15:10:38.933205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:59.918813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075601100000000001,"stat_sum":{"num_bytes":5557804,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":243,"num_read_kb":4111,"num_write":349,"num_write_kb":5610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"36'123","reported_seq":177,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.244288+0000","last_change":"2024-03-28T15:01:31.093706+0000","last_active":"2024-03-28T15:10:01.244288+0000","last_peered":"2024-03-28T15:10:01.244288+0000","last_clean":"2024-03-28T15:10:01.244288+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:10:01.244288+0000","last_undegraded":"2024-03-28T15:10:01.244288+0000","last_fullsized":"2024-03-28T15:10:01.244288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4096,"num_write":123,"num_write_kb":4157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"37'346","reported_seq":2940,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956753+0000","last_change":"2024-03-28T15:04:56.200588+0000","last_active":"2024-03-28T15:10:38.956753+0000","last_peered":"2024-03-28T15:10:38.956753+0000","last_clean":"2024-03-28T15:10:38.956753+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:10:38.956753+0000","last_undegraded":"2024-03-28T15:10:38.956753+0000","last_fullsized":"2024-03-28T15:10:38.956753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:08.686987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038365610000000001,"stat_sum":{"num_bytes":18167101,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":520,"num_read_kb":17729,"num_write":341,"num_write_kb":26114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"37'1696","reported_seq":1753,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:23.448944+0000","last_change":"2024-03-28T15:01:31.095126+0000","last_active":"2024-03-28T15:10:23.448944+0000","last_peered":"2024-03-28T15:10:23.448944+0000","last_clean":"2024-03-28T15:10:23.448944+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:10:23.448944+0000","last_undegraded":"2024-03-28T15:10:23.448944+0000","last_fullsized":"2024-03-28T15:10:23.448944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1696,"log_dups_size":0,"ondisk_log_size":1696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:21.669535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":12343,"num_write":1697,"num_write_kb":9020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"37'339","reported_seq":2633,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.930142+0000","last_change":"2024-03-28T15:04:56.293414+0000","last_active":"2024-03-28T15:10:38.930142+0000","last_peered":"2024-03-28T15:10:38.930142+0000","last_clean":"2024-03-28T15:10:38.930142+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:10:38.930142+0000","last_undegraded":"2024-03-28T15:10:38.930142+0000","last_fullsized":"2024-03-28T15:10:38.930142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:05:58.503942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017181130000000001,"stat_sum":{"num_bytes":6397362,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":220,"num_read_kb":4361,"num_write":330,"num_write_kb":6420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'146","reported_seq":206,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.479981+0000","last_change":"2024-03-28T15:01:31.093197+0000","last_active":"2024-03-28T15:10:38.479981+0000","last_peered":"2024-03-28T15:10:38.479981+0000","last_clean":"2024-03-28T15:10:38.479981+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:10:38.479981+0000","last_undegraded":"2024-03-28T15:10:38.479981+0000","last_fullsized":"2024-03-28T15:10:38.479981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":239,"num_write":363,"num_write_kb":1821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"37'324","reported_seq":3170,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.946211+0000","last_change":"2024-03-28T15:04:56.208109+0000","last_active":"2024-03-28T15:10:38.946211+0000","last_peered":"2024-03-28T15:10:38.946211+0000","last_clean":"2024-03-28T15:10:38.946211+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:10:38.946211+0000","last_undegraded":"2024-03-28T15:10:38.946211+0000","last_fullsized":"2024-03-28T15:10:38.946211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:30.228263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002114524,"stat_sum":{"num_bytes":15625536,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":772,"num_read_kb":10204,"num_write":317,"num_write_kb":19524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"36'175","reported_seq":235,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:14.520739+0000","last_change":"2024-03-28T15:01:31.088204+0000","last_active":"2024-03-28T15:10:14.520739+0000","last_peered":"2024-03-28T15:10:14.520739+0000","last_clean":"2024-03-28T15:10:14.520739+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:10:14.520739+0000","last_undegraded":"2024-03-28T15:10:14.520739+0000","last_fullsized":"2024-03-28T15:10:14.520739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":4117,"num_write":175,"num_write_kb":4205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"37'353","reported_seq":3392,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932088+0000","last_change":"2024-03-28T15:04:56.198120+0000","last_active":"2024-03-28T15:10:38.932088+0000","last_peered":"2024-03-28T15:10:38.932088+0000","last_clean":"2024-03-28T15:10:38.932088+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:10:38.932088+0000","last_undegraded":"2024-03-28T15:10:38.932088+0000","last_fullsized":"2024-03-28T15:10:38.932088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:21.937841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025149030000000002,"stat_sum":{"num_bytes":28522087,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":966,"num_read_kb":19238,"num_write":341,"num_write_kb":28027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'224","reported_seq":279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.644049+0000","last_change":"2024-03-28T15:01:31.095371+0000","last_active":"2024-03-28T15:10:29.644049+0000","last_peered":"2024-03-28T15:10:29.644049+0000","last_clean":"2024-03-28T15:10:29.644049+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:10:29.644049+0000","last_undegraded":"2024-03-28T15:10:29.644049+0000","last_fullsized":"2024-03-28T15:10:29.644049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":230,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"37'351","reported_seq":2841,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:33.541714+0000","last_change":"2024-03-28T15:04:56.196825+0000","last_active":"2024-03-28T15:10:33.541714+0000","last_peered":"2024-03-28T15:10:33.541714+0000","last_clean":"2024-03-28T15:10:33.541714+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:10:33.541714+0000","last_undegraded":"2024-03-28T15:10:33.541714+0000","last_fullsized":"2024-03-28T15:10:33.541714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:43:14.280586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083411100000000001,"stat_sum":{"num_bytes":27575979,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":416,"num_read_kb":10935,"num_write":349,"num_write_kb":27109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"36'18","reported_seq":71,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.860756+0000","last_change":"2024-03-28T15:01:31.094542+0000","last_active":"2024-03-28T15:10:02.860756+0000","last_peered":"2024-03-28T15:10:02.860756+0000","last_clean":"2024-03-28T15:10:02.860756+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:10:02.860756+0000","last_undegraded":"2024-03-28T15:10:02.860756+0000","last_fullsized":"2024-03-28T15:10:02.860756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":18,"num_write_kb":4204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"37'362","reported_seq":3011,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938795+0000","last_change":"2024-03-28T15:04:56.208858+0000","last_active":"2024-03-28T15:10:38.938795+0000","last_peered":"2024-03-28T15:10:38.938795+0000","last_clean":"2024-03-28T15:10:38.938795+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:10:38.938795+0000","last_undegraded":"2024-03-28T15:10:38.938795+0000","last_fullsized":"2024-03-28T15:10:38.938795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:36.687483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015964149999999999,"stat_sum":{"num_bytes":23553308,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":575,"num_read_kb":17168,"num_write":351,"num_write_kb":27278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"37'29","reported_seq":84,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:14.524755+0000","last_change":"2024-03-28T15:01:31.094440+0000","last_active":"2024-03-28T15:10:14.524755+0000","last_peered":"2024-03-28T15:10:14.524755+0000","last_clean":"2024-03-28T15:10:14.524755+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:10:14.524755+0000","last_undegraded":"2024-03-28T15:10:14.524755+0000","last_fullsized":"2024-03-28T15:10:14.524755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":31,"num_write_kb":4146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"37'329","reported_seq":2882,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937387+0000","last_change":"2024-03-28T15:04:56.208307+0000","last_active":"2024-03-28T15:10:38.937387+0000","last_peered":"2024-03-28T15:10:38.937387+0000","last_clean":"2024-03-28T15:10:38.937387+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:10:38.937387+0000","last_undegraded":"2024-03-28T15:10:38.937387+0000","last_fullsized":"2024-03-28T15:10:38.937387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:08.686987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022492050000000002,"stat_sum":{"num_bytes":15497791,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":479,"num_read_kb":3997,"num_write":321,"num_write_kb":15303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"37'214","reported_seq":279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:39.457500+0000","last_change":"2024-03-28T15:01:31.096352+0000","last_active":"2024-03-28T15:10:39.457500+0000","last_peered":"2024-03-28T15:10:39.457500+0000","last_clean":"2024-03-28T15:10:39.457500+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:10:39.457500+0000","last_undegraded":"2024-03-28T15:10:39.457500+0000","last_fullsized":"2024-03-28T15:10:39.457500+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9811737,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":18,"num_read_kb":28682,"num_write":209,"num_write_kb":9709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"37'352","reported_seq":2837,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.930151+0000","last_change":"2024-03-28T15:04:56.534400+0000","last_active":"2024-03-28T15:10:38.930151+0000","last_peered":"2024-03-28T15:10:38.930151+0000","last_clean":"2024-03-28T15:10:38.930151+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:10:38.930151+0000","last_undegraded":"2024-03-28T15:10:38.930151+0000","last_fullsized":"2024-03-28T15:10:38.930151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:43.521019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001719216,"stat_sum":{"num_bytes":11206715,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":411,"num_read_kb":8309,"num_write":347,"num_write_kb":11133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"37'327","reported_seq":2611,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.955085+0000","last_change":"2024-03-28T15:04:56.200097+0000","last_active":"2024-03-28T15:10:38.955085+0000","last_peered":"2024-03-28T15:10:38.955085+0000","last_clean":"2024-03-28T15:10:38.955085+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:10:38.955085+0000","last_undegraded":"2024-03-28T15:10:38.955085+0000","last_fullsized":"2024-03-28T15:10:38.955085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:17:15.635128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036538370000000001,"stat_sum":{"num_bytes":7862984,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":211,"num_read_kb":4119,"num_write":311,"num_write_kb":11936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"26'24","reported_seq":85,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.901412+0000","last_change":"2024-03-28T15:01:31.094281+0000","last_active":"2024-03-28T15:10:01.901412+0000","last_peered":"2024-03-28T15:10:01.901412+0000","last_clean":"2024-03-28T15:10:01.901412+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:10:01.901412+0000","last_undegraded":"2024-03-28T15:10:01.901412+0000","last_fullsized":"2024-03-28T15:10:01.901412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":24,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"37'344","reported_seq":3680,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956194+0000","last_change":"2024-03-28T15:04:56.200342+0000","last_active":"2024-03-28T15:10:38.956194+0000","last_peered":"2024-03-28T15:10:38.956194+0000","last_clean":"2024-03-28T15:10:38.956194+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:10:38.956194+0000","last_undegraded":"2024-03-28T15:10:38.956194+0000","last_fullsized":"2024-03-28T15:10:38.956194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:51:56.779182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037913880000000001,"stat_sum":{"num_bytes":22932535,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":1263,"num_read_kb":23613,"num_write":335,"num_write_kb":26620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"37'170","reported_seq":237,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.643756+0000","last_change":"2024-03-28T15:01:31.093773+0000","last_active":"2024-03-28T15:10:29.643756+0000","last_peered":"2024-03-28T15:10:29.643756+0000","last_clean":"2024-03-28T15:10:29.643756+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:10:29.643756+0000","last_undegraded":"2024-03-28T15:10:29.643756+0000","last_fullsized":"2024-03-28T15:10:29.643756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:54.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":36885,"num_write":163,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"37'319","reported_seq":2904,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.943319+0000","last_change":"2024-03-28T15:04:56.197860+0000","last_active":"2024-03-28T15:10:38.943319+0000","last_peered":"2024-03-28T15:10:38.943319+0000","last_clean":"2024-03-28T15:10:38.943319+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:10:38.943319+0000","last_undegraded":"2024-03-28T15:10:38.943319+0000","last_fullsized":"2024-03-28T15:10:38.943319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:22.085679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023175320000000002,"stat_sum":{"num_bytes":14729150,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":511,"num_read_kb":13471,"num_write":311,"num_write_kb":14544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"37'1099","reported_seq":1154,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.257121+0000","last_change":"2024-03-28T15:01:31.096725+0000","last_active":"2024-03-28T15:10:01.257121+0000","last_peered":"2024-03-28T15:10:01.257121+0000","last_clean":"2024-03-28T15:10:01.257121+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:10:01.257121+0000","last_undegraded":"2024-03-28T15:10:01.257121+0000","last_fullsized":"2024-03-28T15:10:01.257121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":1093,"num_write_kb":5019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"37'346","reported_seq":2920,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935556+0000","last_change":"2024-03-28T15:04:56.207406+0000","last_active":"2024-03-28T15:10:38.935556+0000","last_peered":"2024-03-28T15:10:38.935556+0000","last_clean":"2024-03-28T15:10:38.935556+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:10:38.935556+0000","last_undegraded":"2024-03-28T15:10:38.935556+0000","last_fullsized":"2024-03-28T15:10:38.935556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:57:18.708712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015914119999999999,"stat_sum":{"num_bytes":14776390,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":500,"num_read_kb":7280,"num_write":341,"num_write_kb":14609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"36'104","reported_seq":172,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:03.114633+0000","last_change":"2024-03-28T15:01:31.095351+0000","last_active":"2024-03-28T15:10:03.114633+0000","last_peered":"2024-03-28T15:10:03.114633+0000","last_clean":"2024-03-28T15:10:03.114633+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:10:03.114633+0000","last_undegraded":"2024-03-28T15:10:03.114633+0000","last_fullsized":"2024-03-28T15:10:03.114633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":20500,"num_write":104,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":940897971,"num_objects":10589,"num_object_clones":0,"num_object_copies":21178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10589,"num_whiteouts":0,"num_read":30680,"num_read_kb":666406,"num_write":21173,"num_write_kb":1063512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1935458304,"data_stored":1880255846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21704,"ondisk_log_size":21704,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":182313860,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":492,"num_read_kb":424818,"num_write":6917,"num_write_kb":184630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364691456,"data_stored":364627720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6716,"ondisk_log_size":6716,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574965,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425508,"kb_used_data":336092,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":93946332,"statfs":{"total":96636764160,"available":96201043968,"internally_reserved":0,"allocated":344158208,"data_stored":335751183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":91552202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574966,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294192,"kb_used_data":240808,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94077648,"statfs":{"total":96636764160,"available":96335511552,"internally_reserved":0,"allocated":246587392,"data_stored":241270326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574966,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354032,"kb_used_data":308968,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94017808,"statfs":{"total":96636764160,"available":96274235392,"internally_reserved":0,"allocated":316383232,"data_stored":308090025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574966,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":387780,"kb_used_data":338556,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93984060,"statfs":{"total":96636764160,"available":96239677440,"internally_reserved":0,"allocated":346681344,"data_stored":338235386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574965,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405916,"kb_used_data":360852,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93965924,"statfs":{"total":96636764160,"available":96221106176,"internally_reserved":0,"allocated":369512448,"data_stored":360982077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181040,"kb_used_data":149864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94190800,"statfs":{"total":96636764160,"available":96451379200,"internally_reserved":0,"allocated":153460736,"data_stored":148434971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267092,"kb_used_data":217868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94104748,"statfs":{"total":96636764160,"available":96363261952,"internally_reserved":0,"allocated":223096832,"data_stored":217893056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574965,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368928,"kb_used_data":297560,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94002912,"statfs":{"total":96636764160,"available":96258981888,"internally_reserved":0,"allocated":304701440,"data_stored":296790976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50331808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21508096,"data_stored":21505709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":34099200,"data_stored":34089293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51769344,"data_stored":51754903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50332320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":72732672,"data_stored":72726879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261738496,"data_stored":254075801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172343296,"data_stored":167380119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131538944,"data_stored":126748133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334999552,"data_stored":326711655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":294531072,"data_stored":286332021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":274022400,"data_stored":265965766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":195833856,"data_stored":190756877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":270450688,"data_stored":262285474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:10:45.352 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:10:45.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.487+0000 7fceafb56640 1 Processor -- start 2024-03-28T15:10:45.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.488+0000 7fceafb56640 1 -- start start 2024-03-28T15:10:45.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.488+0000 7fceafb56640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcea80c53c0 0x7fcea80c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.488+0000 7fceafb56640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcea80c57e0 0x7fcea80cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.488+0000 7fceafb56640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcea815b080 con 0x7fcea8154bd0 2024-03-28T15:10:45.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.488+0000 7fceafb56640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fcea80c5ca0 con 0x7fcea80c53c0 2024-03-28T15:10:45.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.488+0000 7fceafb56640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fcea815b360 con 0x7fcea80c57e0 2024-03-28T15:10:45.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.489+0000 7fcead0f1640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcea80c53c0 0x7fcea80c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.489+0000 7fcead8f2640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fcea8154bd0 0x7fcea815a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:57510/0 (socket says 172.21.15.186:57510) 2024-03-28T15:10:45.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.489+0000 7fceae0f3640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcea80c57e0 0x7fcea80cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.489+0000 7fcead0f1640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcea80c53c0 0x7fcea80c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:43162/0 (socket says 172.21.15.186:43162) 2024-03-28T15:10:45.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.489+0000 7fcead8f2640 1 -- 172.21.15.186:0/3554433378 learned_addr learned my addr 172.21.15.186:0/3554433378 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:45.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.489+0000 7fceae0f3640 1 -- 172.21.15.186:0/3554433378 >> v2:172.21.15.191:3300/0 conn(0x7fcea80c53c0 msgr2=0x7fcea80c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:45.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.489+0000 7fceae0f3640 1 --2- 172.21.15.186:0/3554433378 >> v2:172.21.15.191:3300/0 conn(0x7fcea80c53c0 0x7fcea80c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.489+0000 7fceae0f3640 1 -- 172.21.15.186:0/3554433378 >> v1:172.21.15.191:6789/0 conn(0x7fcea8154bd0 legacy=0x7fcea815a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.489+0000 7fceae0f3640 1 -- 172.21.15.186:0/3554433378 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea815b640 con 0x7fcea80c57e0 2024-03-28T15:10:45.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.489+0000 7fcead0f1640 1 --2- 172.21.15.186:0/3554433378 >> v2:172.21.15.191:3300/0 conn(0x7fcea80c53c0 0x7fcea80c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:45.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.490+0000 7fceae0f3640 1 --2- 172.21.15.186:0/3554433378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcea80c57e0 0x7fcea80cb040 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009d90 tx=0x7fcea002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=885f3d802804ed5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:45.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.490+0000 7fceac8f0640 1 -- 172.21.15.186:0/3554433378 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0039070 con 0x7fcea80c57e0 2024-03-28T15:10:45.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.490+0000 7fceac8f0640 1 -- 172.21.15.186:0/3554433378 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea002cce0 con 0x7fcea80c57e0 2024-03-28T15:10:45.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.490+0000 7fceac8f0640 1 -- 172.21.15.186:0/3554433378 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0034660 con 0x7fcea80c57e0 2024-03-28T15:10:45.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.490+0000 7fceafb56640 1 -- 172.21.15.186:0/3554433378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcea80c57e0 msgr2=0x7fcea80cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.490+0000 7fceafb56640 1 --2- 172.21.15.186:0/3554433378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcea80c57e0 0x7fcea80cb040 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009d90 tx=0x7fcea002bf30 comp rx=0 tx=0).stop 2024-03-28T15:10:45.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.490+0000 7fceafb56640 1 -- 172.21.15.186:0/3554433378 shutdown_connections 2024-03-28T15:10:45.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.490+0000 7fceafb56640 1 --2- 172.21.15.186:0/3554433378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcea80c57e0 0x7fcea80cb040 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.490+0000 7fceafb56640 1 --2- 172.21.15.186:0/3554433378 >> v2:172.21.15.191:3300/0 conn(0x7fcea80c53c0 0x7fcea80c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.490+0000 7fceafb56640 1 -- 172.21.15.186:0/3554433378 >> 172.21.15.186:0/3554433378 conn(0x7fcea80731f0 msgr2=0x7fcea8073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:45.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.490+0000 7fceafb56640 1 -- 172.21.15.186:0/3554433378 shutdown_connections 2024-03-28T15:10:45.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.491+0000 7fceafb56640 1 -- 172.21.15.186:0/3554433378 wait complete. 2024-03-28T15:10:45.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fceafb56640 1 Processor -- start 2024-03-28T15:10:45.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fceafb56640 1 -- start start 2024-03-28T15:10:45.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fceafb56640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcea80c53c0 0x7fcea81f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fceafb56640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcea80c57e0 0x7fcea81f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fceafb56640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcea8154bd0 0x7fcea81f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fceafb56640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcea80cc390 con 0x7fcea8154bd0 2024-03-28T15:10:45.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fceafb56640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcea80c61d0 con 0x7fcea80c53c0 2024-03-28T15:10:45.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fceafb56640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fcea80c6c30 con 0x7fcea80c57e0 2024-03-28T15:10:45.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fceae0f3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcea8154bd0 0x7fcea81f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fcead8f2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcea80c53c0 0x7fcea81f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fceae0f3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcea8154bd0 0x7fcea81f2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36650/0 (socket says 172.21.15.186:36650) 2024-03-28T15:10:45.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fceae0f3640 1 -- 172.21.15.186:0/3302087613 learned_addr learned my addr 172.21.15.186:0/3302087613 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:45.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.492+0000 7fcead0f1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcea80c57e0 0x7fcea81f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.493+0000 7fceae0f3640 1 -- 172.21.15.186:0/3302087613 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcea80c57e0 msgr2=0x7fcea81f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.493+0000 7fceae0f3640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcea80c57e0 0x7fcea81f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.493+0000 7fceae0f3640 1 -- 172.21.15.186:0/3302087613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcea80c53c0 msgr2=0x7fcea81f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.493+0000 7fceae0f3640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcea80c53c0 0x7fcea81f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.493+0000 7fceae0f3640 1 -- 172.21.15.186:0/3302087613 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea8155680 con 0x7fcea8154bd0 2024-03-28T15:10:45.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.493+0000 7fceae0f3640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcea8154bd0 0x7fcea81f2510 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009eb0 tx=0x7fcea0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:45.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.493+0000 7fce9a7fc640 1 -- 172.21.15.186:0/3302087613 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0039050 con 0x7fcea8154bd0 2024-03-28T15:10:45.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.493+0000 7fce9a7fc640 1 -- 172.21.15.186:0/3302087613 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea0045070 con 0x7fcea8154bd0 2024-03-28T15:10:45.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.493+0000 7fce9a7fc640 1 -- 172.21.15.186:0/3302087613 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea003e830 con 0x7fcea8154bd0 2024-03-28T15:10:45.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.493+0000 7fceafb56640 1 -- 172.21.15.186:0/3302087613 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea8156ea0 con 0x7fcea8154bd0 2024-03-28T15:10:45.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.493+0000 7fceafb56640 1 -- 172.21.15.186:0/3302087613 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea81553a0 con 0x7fcea8154bd0 2024-03-28T15:10:45.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.494+0000 7fceafb56640 1 -- 172.21.15.186:0/3302087613 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea8069500 con 0x7fcea8154bd0 2024-03-28T15:10:45.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.495+0000 7fce9a7fc640 1 -- 172.21.15.186:0/3302087613 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fcea003e9d0 con 0x7fcea8154bd0 2024-03-28T15:10:45.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.495+0000 7fce9a7fc640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fce8c06c560 0x7fce8c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.495+0000 7fce9a7fc640 1 -- 172.21.15.186:0/3302087613 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fcea00ae690 con 0x7fcea8154bd0 2024-03-28T15:10:45.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.495+0000 7fcead8f2640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fce8c06c560 0x7fce8c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.495+0000 7fcead8f2640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fce8c06c560 0x7fce8c06ea20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fce9c007fd0 tx=0x7fce9c007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:45.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.499+0000 7fce9a7fc640 1 -- 172.21.15.186:0/3302087613 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fcea007c4e0 con 0x7fcea8154bd0 2024-03-28T15:10:45.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.671+0000 7fceafb56640 1 -- 172.21.15.186:0/3302087613 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcea80b2ce0 con 0x7fce8c06c560 2024-03-28T15:10:45.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.677+0000 7fce9a7fc640 1 -- 172.21.15.186:0/3302087613 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+256109 (secure 0 0 0) 0x7fcea80b2ce0 con 0x7fce8c06c560 2024-03-28T15:10:45.681 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:45.682 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:10:45.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.680+0000 7fceafb56640 1 -- 172.21.15.186:0/3302087613 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fce8c06c560 msgr2=0x7fce8c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.680+0000 7fceafb56640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fce8c06c560 0x7fce8c06ea20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fce9c007fd0 tx=0x7fce9c007e80 comp rx=0 tx=0).stop 2024-03-28T15:10:45.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.680+0000 7fceafb56640 1 -- 172.21.15.186:0/3302087613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcea8154bd0 msgr2=0x7fcea81f2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.680+0000 7fceafb56640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcea8154bd0 0x7fcea81f2510 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009eb0 tx=0x7fcea0004a40 comp rx=0 tx=0).stop 2024-03-28T15:10:45.684 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.680+0000 7fceafb56640 1 -- 172.21.15.186:0/3302087613 shutdown_connections 2024-03-28T15:10:45.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.680+0000 7fceafb56640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fce8c06c560 0x7fce8c06ea20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.680+0000 7fceafb56640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcea8154bd0 0x7fcea81f2510 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.680+0000 7fceafb56640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcea80c57e0 0x7fcea81f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.680+0000 7fceafb56640 1 --2- 172.21.15.186:0/3302087613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcea80c53c0 0x7fcea81f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.680+0000 7fceafb56640 1 -- 172.21.15.186:0/3302087613 >> 172.21.15.186:0/3302087613 conn(0x7fcea80731f0 msgr2=0x7fcea81e3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:45.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.681+0000 7fceafb56640 1 -- 172.21.15.186:0/3302087613 shutdown_connections 2024-03-28T15:10:45.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.681+0000 7fceafb56640 1 -- 172.21.15.186:0/3302087613 wait complete. 2024-03-28T15:10:45.692 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-03-28T15:10:44.750496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1123802199,"num_objects":10721,"num_object_clones":0,"num_object_copies":21442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10721,"num_whiteouts":0,"num_read":31288,"num_read_kb":1091324,"num_write":28249,"num_write_kb":1250558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28537,"ondisk_log_size":28537,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2684488,"kb_used_data":2250568,"kb_used_omap":12,"kb_used_meta":433843,"kb_avail":752290232,"statfs":{"total":773094113280,"available":770345197568,"internally_reserved":0,"allocated":2304581632,"data_stored":2247448000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":444255793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26311273,"num_objects":1458,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":6,"num_read_kb":908,"num_write":1684,"num_write_kb":27163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003096"},"pg_stats":[{"pgid":"3.3f","version":"37'335","reported_seq":3248,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.950296+0000","last_change":"2024-03-28T15:04:56.745680+0000","last_active":"2024-03-28T15:10:38.950296+0000","last_peered":"2024-03-28T15:10:38.950296+0000","last_clean":"2024-03-28T15:10:38.950296+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:10:38.950296+0000","last_undegraded":"2024-03-28T15:10:38.950296+0000","last_fullsized":"2024-03-28T15:10:38.950296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:21:41.971527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024885649999999999,"stat_sum":{"num_bytes":26509954,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":839,"num_read_kb":12103,"num_write":327,"num_write_kb":26054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"37'316","reported_seq":2406,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.930520+0000","last_change":"2024-03-28T15:04:56.294141+0000","last_active":"2024-03-28T15:10:38.930520+0000","last_peered":"2024-03-28T15:10:38.930520+0000","last_clean":"2024-03-28T15:10:38.930520+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:10:38.930520+0000","last_undegraded":"2024-03-28T15:10:38.930520+0000","last_fullsized":"2024-03-28T15:10:38.930520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:43.533807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019334860000000001,"stat_sum":{"num_bytes":1345083,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":314,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"37'329","reported_seq":3088,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.922289+0000","last_change":"2024-03-28T15:04:56.198079+0000","last_active":"2024-03-28T15:10:38.922289+0000","last_peered":"2024-03-28T15:10:38.922289+0000","last_clean":"2024-03-28T15:10:38.922289+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:10:38.922289+0000","last_undegraded":"2024-03-28T15:10:38.922289+0000","last_fullsized":"2024-03-28T15:10:38.922289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:00:42.406838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025151280000000002,"stat_sum":{"num_bytes":12619399,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":685,"num_read_kb":17683,"num_write":321,"num_write_kb":20675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"37'314","reported_seq":3077,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938235+0000","last_change":"2024-03-28T15:04:56.747212+0000","last_active":"2024-03-28T15:10:38.938235+0000","last_peered":"2024-03-28T15:10:38.938235+0000","last_clean":"2024-03-28T15:10:38.938235+0000","last_became_active":"2024-03-28T15:02:12.485356+0000","last_became_peered":"2024-03-28T15:02:12.485356+0000","last_unstale":"2024-03-28T15:10:38.938235+0000","last_undegraded":"2024-03-28T15:10:38.938235+0000","last_fullsized":"2024-03-28T15:10:38.938235+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:51:41.708849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014203849999999999,"stat_sum":{"num_bytes":19355212,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":695,"num_read_kb":14278,"num_write":306,"num_write_kb":19012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"37'363","reported_seq":2551,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938805+0000","last_change":"2024-03-28T15:04:56.745133+0000","last_active":"2024-03-28T15:10:38.938805+0000","last_peered":"2024-03-28T15:10:38.938805+0000","last_clean":"2024-03-28T15:10:38.938805+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:10:38.938805+0000","last_undegraded":"2024-03-28T15:10:38.938805+0000","last_fullsized":"2024-03-28T15:10:38.938805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:21.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002796064,"stat_sum":{"num_bytes":10679980,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":115,"num_read_kb":4114,"num_write":361,"num_write_kb":10554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"37'327","reported_seq":2639,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.946373+0000","last_change":"2024-03-28T15:04:56.200817+0000","last_active":"2024-03-28T15:10:38.946373+0000","last_peered":"2024-03-28T15:10:38.946373+0000","last_clean":"2024-03-28T15:10:38.946373+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:10:38.946373+0000","last_undegraded":"2024-03-28T15:10:38.946373+0000","last_fullsized":"2024-03-28T15:10:38.946373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:03:09.771351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003798718,"stat_sum":{"num_bytes":10128543,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":239,"num_read_kb":4123,"num_write":321,"num_write_kb":10056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"37'354","reported_seq":3455,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.950756+0000","last_change":"2024-03-28T15:04:56.743147+0000","last_active":"2024-03-28T15:10:38.950756+0000","last_peered":"2024-03-28T15:10:38.950756+0000","last_clean":"2024-03-28T15:10:38.950756+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:10:38.950756+0000","last_undegraded":"2024-03-28T15:10:38.950756+0000","last_fullsized":"2024-03-28T15:10:38.950756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:20.151357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001484766,"stat_sum":{"num_bytes":23645872,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":1028,"num_read_kb":17521,"num_write":350,"num_write_kb":23210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"37'334","reported_seq":2944,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.959962+0000","last_change":"2024-03-28T15:04:56.208811+0000","last_active":"2024-03-28T15:10:38.959962+0000","last_peered":"2024-03-28T15:10:38.959962+0000","last_clean":"2024-03-28T15:10:38.959962+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:10:38.959962+0000","last_undegraded":"2024-03-28T15:10:38.959962+0000","last_fullsized":"2024-03-28T15:10:38.959962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:57.578217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017148910000000001,"stat_sum":{"num_bytes":19312172,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":537,"num_read_kb":7179,"num_write":326,"num_write_kb":18943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"37'330","reported_seq":2712,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960654+0000","last_change":"2024-03-28T15:04:56.745714+0000","last_active":"2024-03-28T15:10:38.960654+0000","last_peered":"2024-03-28T15:10:38.960654+0000","last_clean":"2024-03-28T15:10:38.960654+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:10:38.960654+0000","last_undegraded":"2024-03-28T15:10:38.960654+0000","last_fullsized":"2024-03-28T15:10:38.960654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:27:32.222447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023275240000000001,"stat_sum":{"num_bytes":6398085,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":309,"num_read_kb":8863,"num_write":320,"num_write_kb":10489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"37'327","reported_seq":3210,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.961682+0000","last_change":"2024-03-28T15:04:56.534451+0000","last_active":"2024-03-28T15:10:38.961682+0000","last_peered":"2024-03-28T15:10:38.961682+0000","last_clean":"2024-03-28T15:10:38.961682+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:10:38.961682+0000","last_undegraded":"2024-03-28T15:10:38.961682+0000","last_fullsized":"2024-03-28T15:10:38.961682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:43.533807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019348060000000001,"stat_sum":{"num_bytes":19157707,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":810,"num_read_kb":17508,"num_write":311,"num_write_kb":18862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"37'303","reported_seq":3011,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.959082+0000","last_change":"2024-03-28T15:04:56.534170+0000","last_active":"2024-03-28T15:10:38.959082+0000","last_peered":"2024-03-28T15:10:38.959082+0000","last_clean":"2024-03-28T15:10:38.959082+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:10:38.959082+0000","last_undegraded":"2024-03-28T15:10:38.959082+0000","last_fullsized":"2024-03-28T15:10:38.959082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:20.495757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018723170000000001,"stat_sum":{"num_bytes":18249049,"num_objects":129,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":635,"num_read_kb":13018,"num_write":297,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"37'296","reported_seq":2603,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.934653+0000","last_change":"2024-03-28T15:04:56.199593+0000","last_active":"2024-03-28T15:10:38.934653+0000","last_peered":"2024-03-28T15:10:38.934653+0000","last_clean":"2024-03-28T15:10:38.934653+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:10:38.934653+0000","last_undegraded":"2024-03-28T15:10:38.934653+0000","last_fullsized":"2024-03-28T15:10:38.934653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:50.527037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013925400000000001,"stat_sum":{"num_bytes":6129603,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":234,"num_read_kb":4757,"num_write":288,"num_write_kb":10236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"37'297","reported_seq":2859,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936011+0000","last_change":"2024-03-28T15:04:56.207750+0000","last_active":"2024-03-28T15:10:38.936011+0000","last_peered":"2024-03-28T15:10:38.936011+0000","last_clean":"2024-03-28T15:10:38.936011+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:10:38.936011+0000","last_undegraded":"2024-03-28T15:10:38.936011+0000","last_fullsized":"2024-03-28T15:10:38.936011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:17:15.635128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017572180000000001,"stat_sum":{"num_bytes":15029757,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":489,"num_read_kb":12548,"num_write":286,"num_write_kb":23016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"37'341","reported_seq":2613,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932033+0000","last_change":"2024-03-28T15:04:56.197644+0000","last_active":"2024-03-28T15:10:38.932033+0000","last_peered":"2024-03-28T15:10:38.932033+0000","last_clean":"2024-03-28T15:10:38.932033+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:10:38.932033+0000","last_undegraded":"2024-03-28T15:10:38.932033+0000","last_fullsized":"2024-03-28T15:10:38.932033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:08.587017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002253971,"stat_sum":{"num_bytes":6664925,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":199,"num_read_kb":4930,"num_write":335,"num_write_kb":6638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"37'357","reported_seq":3528,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.926788+0000","last_change":"2024-03-28T15:04:56.293889+0000","last_active":"2024-03-28T15:10:38.926788+0000","last_peered":"2024-03-28T15:10:38.926788+0000","last_clean":"2024-03-28T15:10:38.926788+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:10:38.926788+0000","last_undegraded":"2024-03-28T15:10:38.926788+0000","last_fullsized":"2024-03-28T15:10:38.926788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:20.495757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179673,"stat_sum":{"num_bytes":23044368,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":1098,"num_read_kb":19203,"num_write":350,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"37'357","reported_seq":2912,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.945394+0000","last_change":"2024-03-28T15:04:56.200579+0000","last_active":"2024-03-28T15:10:38.945394+0000","last_peered":"2024-03-28T15:10:38.945394+0000","last_clean":"2024-03-28T15:10:38.945394+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:10:38.945394+0000","last_undegraded":"2024-03-28T15:10:38.945394+0000","last_fullsized":"2024-03-28T15:10:38.945394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:30.228263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039634529999999996,"stat_sum":{"num_bytes":16147473,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":482,"num_read_kb":8871,"num_write":348,"num_write_kb":19203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"37'349","reported_seq":2573,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960198+0000","last_change":"2024-03-28T15:04:56.201087+0000","last_active":"2024-03-28T15:10:38.960198+0000","last_peered":"2024-03-28T15:10:38.960198+0000","last_clean":"2024-03-28T15:10:38.960198+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:10:38.960198+0000","last_undegraded":"2024-03-28T15:10:38.960198+0000","last_fullsized":"2024-03-28T15:10:38.960198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:36.687483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027424319999999999,"stat_sum":{"num_bytes":10498072,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":151,"num_read_kb":9005,"num_write":340,"num_write_kb":14498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"37'311","reported_seq":2414,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936309+0000","last_change":"2024-03-28T15:04:56.746916+0000","last_active":"2024-03-28T15:10:38.936309+0000","last_peered":"2024-03-28T15:10:38.936309+0000","last_clean":"2024-03-28T15:10:38.936309+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:10:38.936309+0000","last_undegraded":"2024-03-28T15:10:38.936309+0000","last_fullsized":"2024-03-28T15:10:38.936309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:20:35.393196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179399,"stat_sum":{"num_bytes":9508598,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":30,"num_read_kb":42,"num_write":300,"num_write_kb":9445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"37'324","reported_seq":2710,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939190+0000","last_change":"2024-03-28T15:04:56.745421+0000","last_active":"2024-03-28T15:10:38.939190+0000","last_peered":"2024-03-28T15:10:38.939190+0000","last_clean":"2024-03-28T15:10:38.939190+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:10:38.939190+0000","last_undegraded":"2024-03-28T15:10:38.939190+0000","last_fullsized":"2024-03-28T15:10:38.939190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:50.382005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030062439999999999,"stat_sum":{"num_bytes":14185939,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":312,"num_read_kb":12813,"num_write":317,"num_write_kb":18119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"37'356","reported_seq":3288,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937246+0000","last_change":"2024-03-28T15:04:56.534664+0000","last_active":"2024-03-28T15:10:38.937246+0000","last_peered":"2024-03-28T15:10:38.937246+0000","last_clean":"2024-03-28T15:10:38.937246+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:10:38.937246+0000","last_undegraded":"2024-03-28T15:10:38.937246+0000","last_fullsized":"2024-03-28T15:10:38.937246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:10.614269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001798584,"stat_sum":{"num_bytes":30036364,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":858,"num_read_kb":15492,"num_write":348,"num_write_kb":37706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"37'342","reported_seq":2724,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957518+0000","last_change":"2024-03-28T15:04:56.198376+0000","last_active":"2024-03-28T15:10:38.957518+0000","last_peered":"2024-03-28T15:10:38.957518+0000","last_clean":"2024-03-28T15:10:38.957518+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:10:38.957518+0000","last_undegraded":"2024-03-28T15:10:38.957518+0000","last_fullsized":"2024-03-28T15:10:38.957518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:55:39.836954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025216930000000002,"stat_sum":{"num_bytes":10421413,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":308,"num_read_kb":8356,"num_write":331,"num_write_kb":10350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"37'337","reported_seq":2523,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935072+0000","last_change":"2024-03-28T15:04:56.746844+0000","last_active":"2024-03-28T15:10:38.935072+0000","last_peered":"2024-03-28T15:10:38.935072+0000","last_clean":"2024-03-28T15:10:38.935072+0000","last_became_active":"2024-03-28T15:02:12.178196+0000","last_became_peered":"2024-03-28T15:02:12.178196+0000","last_unstale":"2024-03-28T15:10:38.935072+0000","last_undegraded":"2024-03-28T15:10:38.935072+0000","last_fullsized":"2024-03-28T15:10:38.935072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:59:11.409442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001195878,"stat_sum":{"num_bytes":9431253,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":119,"num_read_kb":4113,"num_write":327,"num_write_kb":13479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"37'331","reported_seq":2998,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939818+0000","last_change":"2024-03-28T15:04:56.207990+0000","last_active":"2024-03-28T15:10:38.939818+0000","last_peered":"2024-03-28T15:10:38.939818+0000","last_clean":"2024-03-28T15:10:38.939818+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:10:38.939818+0000","last_undegraded":"2024-03-28T15:10:38.939818+0000","last_fullsized":"2024-03-28T15:10:38.939818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:51:56.779182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020004409999999999,"stat_sum":{"num_bytes":16319296,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":593,"num_read_kb":18222,"num_write":323,"num_write_kb":20203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"37'356","reported_seq":2799,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956957+0000","last_change":"2024-03-28T15:04:56.200922+0000","last_active":"2024-03-28T15:10:38.956957+0000","last_peered":"2024-03-28T15:10:38.956957+0000","last_clean":"2024-03-28T15:10:38.956957+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:10:38.956957+0000","last_undegraded":"2024-03-28T15:10:38.956957+0000","last_fullsized":"2024-03-28T15:10:38.956957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:57.578217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031981639999999999,"stat_sum":{"num_bytes":14893759,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":369,"num_read_kb":10097,"num_write":349,"num_write_kb":14727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"37'314","reported_seq":3255,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.931560+0000","last_change":"2024-03-28T15:04:56.207665+0000","last_active":"2024-03-28T15:10:38.931560+0000","last_peered":"2024-03-28T15:10:38.931560+0000","last_clean":"2024-03-28T15:10:38.931560+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:10:38.931560+0000","last_undegraded":"2024-03-28T15:10:38.931560+0000","last_fullsized":"2024-03-28T15:10:38.931560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:36:17.765596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001813119,"stat_sum":{"num_bytes":18699537,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":867,"num_read_kb":15099,"num_write":310,"num_write_kb":18421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"37'377","reported_seq":2618,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960107+0000","last_change":"2024-03-28T15:04:56.534606+0000","last_active":"2024-03-28T15:10:38.960107+0000","last_peered":"2024-03-28T15:10:38.960107+0000","last_clean":"2024-03-28T15:10:38.960107+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:10:38.960107+0000","last_undegraded":"2024-03-28T15:10:38.960107+0000","last_fullsized":"2024-03-28T15:10:38.960107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:23:24.123040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019272180000000001,"stat_sum":{"num_bytes":11206818,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":167,"num_read_kb":4115,"num_write":363,"num_write_kb":11124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"37'320","reported_seq":2923,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936502+0000","last_change":"2024-03-28T15:04:56.294225+0000","last_active":"2024-03-28T15:10:38.936502+0000","last_peered":"2024-03-28T15:10:38.936502+0000","last_clean":"2024-03-28T15:10:38.936502+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:10:38.936502+0000","last_undegraded":"2024-03-28T15:10:38.936502+0000","last_fullsized":"2024-03-28T15:10:38.936502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:23:24.123040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019373070000000001,"stat_sum":{"num_bytes":22270707,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":529,"num_read_kb":12598,"num_write":314,"num_write_kb":21911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"37'315","reported_seq":3227,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.942957+0000","last_change":"2024-03-28T15:04:56.208607+0000","last_active":"2024-03-28T15:10:38.942957+0000","last_peered":"2024-03-28T15:10:38.942957+0000","last_clean":"2024-03-28T15:10:38.942957+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:10:38.942957+0000","last_undegraded":"2024-03-28T15:10:38.942957+0000","last_fullsized":"2024-03-28T15:10:38.942957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:03:09.771351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020980170000000002,"stat_sum":{"num_bytes":17909578,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":838,"num_read_kb":18539,"num_write":311,"num_write_kb":20928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"37'358","reported_seq":3192,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935632+0000","last_change":"2024-03-28T15:04:56.198529+0000","last_active":"2024-03-28T15:10:38.935632+0000","last_peered":"2024-03-28T15:10:38.935632+0000","last_clean":"2024-03-28T15:10:38.935632+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:10:38.935632+0000","last_undegraded":"2024-03-28T15:10:38.935632+0000","last_fullsized":"2024-03-28T15:10:38.935632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:41:01.927661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026139499999999999,"stat_sum":{"num_bytes":26288347,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":760,"num_read_kb":8154,"num_write":350,"num_write_kb":25854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"37'359","reported_seq":2900,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956962+0000","last_change":"2024-03-28T15:04:56.200966+0000","last_active":"2024-03-28T15:10:38.956962+0000","last_peered":"2024-03-28T15:10:38.956962+0000","last_clean":"2024-03-28T15:10:38.956962+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:10:38.956962+0000","last_undegraded":"2024-03-28T15:10:38.956962+0000","last_fullsized":"2024-03-28T15:10:38.956962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:37.204800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036450829999999999,"stat_sum":{"num_bytes":10387544,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":467,"num_read_kb":8853,"num_write":352,"num_write_kb":10266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"37'386","reported_seq":2968,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.942998+0000","last_change":"2024-03-28T15:04:56.747393+0000","last_active":"2024-03-28T15:10:38.942998+0000","last_peered":"2024-03-28T15:10:38.942998+0000","last_clean":"2024-03-28T15:10:38.942998+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:10:38.942998+0000","last_undegraded":"2024-03-28T15:10:38.942998+0000","last_fullsized":"2024-03-28T15:10:38.942998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:39:11.217145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001020962,"stat_sum":{"num_bytes":12939697,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":508,"num_read_kb":8898,"num_write":372,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"37'384","reported_seq":2787,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932311+0000","last_change":"2024-03-28T15:04:56.534204+0000","last_active":"2024-03-28T15:10:38.932311+0000","last_peered":"2024-03-28T15:10:38.932311+0000","last_clean":"2024-03-28T15:10:38.932311+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:10:38.932311+0000","last_undegraded":"2024-03-28T15:10:38.932311+0000","last_fullsized":"2024-03-28T15:10:38.932311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:56:09.141610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019055229999999999,"stat_sum":{"num_bytes":11304826,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":329,"num_read_kb":5287,"num_write":374,"num_write_kb":11236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"37'322","reported_seq":2854,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.921473+0000","last_change":"2024-03-28T15:04:56.196629+0000","last_active":"2024-03-28T15:10:38.921473+0000","last_peered":"2024-03-28T15:10:38.921473+0000","last_clean":"2024-03-28T15:10:38.921473+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:10:38.921473+0000","last_undegraded":"2024-03-28T15:10:38.921473+0000","last_fullsized":"2024-03-28T15:10:38.921473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:51.357070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078031800000000001,"stat_sum":{"num_bytes":14482293,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":459,"num_read_kb":10396,"num_write":315,"num_write_kb":14305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"36'18","reported_seq":74,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.776351+0000","last_change":"2024-03-28T15:01:31.097415+0000","last_active":"2024-03-28T15:09:56.776351+0000","last_peered":"2024-03-28T15:09:56.776351+0000","last_clean":"2024-03-28T15:09:56.776351+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:09:56.776351+0000","last_undegraded":"2024-03-28T15:09:56.776351+0000","last_fullsized":"2024-03-28T15:09:56.776351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":16384,"num_write":21,"num_write_kb":4230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"37'325","reported_seq":2638,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.934802+0000","last_change":"2024-03-28T15:04:56.198560+0000","last_active":"2024-03-28T15:10:38.934802+0000","last_peered":"2024-03-28T15:10:38.934802+0000","last_clean":"2024-03-28T15:10:38.934802+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:10:38.934802+0000","last_undegraded":"2024-03-28T15:10:38.934802+0000","last_fullsized":"2024-03-28T15:10:38.934802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:51.739886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023690579999999998,"stat_sum":{"num_bytes":11043752,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":240,"num_read_kb":4317,"num_write":317,"num_write_kb":10950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"37'647","reported_seq":711,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.472792+0000","last_change":"2024-03-28T15:01:31.088437+0000","last_active":"2024-03-28T15:10:02.472792+0000","last_peered":"2024-03-28T15:10:02.472792+0000","last_clean":"2024-03-28T15:10:02.472792+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:10:02.472792+0000","last_undegraded":"2024-03-28T15:10:02.472792+0000","last_fullsized":"2024-03-28T15:10:02.472792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":34526,"num_write":642,"num_write_kb":12757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"36'58","reported_seq":112,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.775891+0000","last_change":"2024-03-28T15:01:31.097499+0000","last_active":"2024-03-28T15:09:56.775891+0000","last_peered":"2024-03-28T15:09:56.775891+0000","last_clean":"2024-03-28T15:09:56.775891+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:09:56.775891+0000","last_undegraded":"2024-03-28T15:09:56.775891+0000","last_fullsized":"2024-03-28T15:09:56.775891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":62,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"37'402","reported_seq":3693,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936725+0000","last_change":"2024-03-28T15:04:56.208655+0000","last_active":"2024-03-28T15:10:38.936725+0000","last_peered":"2024-03-28T15:10:38.936725+0000","last_clean":"2024-03-28T15:10:38.936725+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:10:38.936725+0000","last_undegraded":"2024-03-28T15:10:38.936725+0000","last_fullsized":"2024-03-28T15:10:38.936725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:37.204800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017492600000000001,"stat_sum":{"num_bytes":22338843,"num_objects":188,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":1218,"num_read_kb":22454,"num_write":394,"num_write_kb":30214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"37'604","reported_seq":661,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:30.364435+0000","last_change":"2024-03-28T15:01:31.086252+0000","last_active":"2024-03-28T15:10:30.364435+0000","last_peered":"2024-03-28T15:10:30.364435+0000","last_clean":"2024-03-28T15:10:30.364435+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:10:30.364435+0000","last_undegraded":"2024-03-28T15:10:30.364435+0000","last_fullsized":"2024-03-28T15:10:30.364435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":603,"num_write_kb":12492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"37'341","reported_seq":3347,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937426+0000","last_change":"2024-03-28T15:04:56.293936+0000","last_active":"2024-03-28T15:10:38.937426+0000","last_peered":"2024-03-28T15:10:38.937426+0000","last_clean":"2024-03-28T15:10:38.937426+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:10:38.937426+0000","last_undegraded":"2024-03-28T15:10:38.937426+0000","last_fullsized":"2024-03-28T15:10:38.937426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:03.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022536280000000001,"stat_sum":{"num_bytes":19647424,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":933,"num_read_kb":22521,"num_write":330,"num_write_kb":23406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"37'5","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.246758+0000","last_change":"2024-03-28T15:01:31.092338+0000","last_active":"2024-03-28T15:09:56.246758+0000","last_peered":"2024-03-28T15:09:56.246758+0000","last_clean":"2024-03-28T15:09:56.246758+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:09:56.246758+0000","last_undegraded":"2024-03-28T15:09:56.246758+0000","last_fullsized":"2024-03-28T15:09:56.246758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"37'366","reported_seq":3228,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960208+0000","last_change":"2024-03-28T15:04:56.200271+0000","last_active":"2024-03-28T15:10:38.960208+0000","last_peered":"2024-03-28T15:10:38.960208+0000","last_clean":"2024-03-28T15:10:38.960208+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:10:38.960208+0000","last_undegraded":"2024-03-28T15:10:38.960208+0000","last_fullsized":"2024-03-28T15:10:38.960208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:36:17.765596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043389099999999996,"stat_sum":{"num_bytes":22872414,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":789,"num_read_kb":19564,"num_write":357,"num_write_kb":22493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"37'70","reported_seq":124,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.349608+0000","last_change":"2024-03-28T15:01:31.088909+0000","last_active":"2024-03-28T15:10:02.349608+0000","last_peered":"2024-03-28T15:10:02.349608+0000","last_clean":"2024-03-28T15:10:02.349608+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:10:02.349608+0000","last_undegraded":"2024-03-28T15:10:02.349608+0000","last_fullsized":"2024-03-28T15:10:02.349608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":66,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"37'323","reported_seq":2969,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.943307+0000","last_change":"2024-03-28T15:04:56.197873+0000","last_active":"2024-03-28T15:10:38.943307+0000","last_peered":"2024-03-28T15:10:38.943307+0000","last_clean":"2024-03-28T15:10:38.943307+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:10:38.943307+0000","last_undegraded":"2024-03-28T15:10:38.943307+0000","last_fullsized":"2024-03-28T15:10:38.943307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:52.072532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023945189999999999,"stat_sum":{"num_bytes":10186807,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":573,"num_read_kb":8919,"num_write":314,"num_write_kb":10118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"36'193","reported_seq":257,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.698311+0000","last_change":"2024-03-28T15:01:31.095935+0000","last_active":"2024-03-28T15:10:02.698311+0000","last_peered":"2024-03-28T15:10:02.698311+0000","last_clean":"2024-03-28T15:10:02.698311+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:10:02.698311+0000","last_undegraded":"2024-03-28T15:10:02.698311+0000","last_fullsized":"2024-03-28T15:10:02.698311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":4122,"num_write":194,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"37'321","reported_seq":3146,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957720+0000","last_change":"2024-03-28T15:04:56.199307+0000","last_active":"2024-03-28T15:10:38.957720+0000","last_peered":"2024-03-28T15:10:38.957720+0000","last_clean":"2024-03-28T15:10:38.957720+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:10:38.957720+0000","last_undegraded":"2024-03-28T15:10:38.957720+0000","last_fullsized":"2024-03-28T15:10:38.957720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:32:03.268809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254104,"stat_sum":{"num_bytes":14752712,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":752,"num_read_kb":16498,"num_write":310,"num_write_kb":18664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"37'122","reported_seq":201,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.389422+0000","last_change":"2024-03-28T15:01:31.089380+0000","last_active":"2024-03-28T15:10:29.389422+0000","last_peered":"2024-03-28T15:10:29.389422+0000","last_clean":"2024-03-28T15:10:29.389422+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:10:29.389422+0000","last_undegraded":"2024-03-28T15:10:29.389422+0000","last_fullsized":"2024-03-28T15:10:29.389422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32804,"num_write":115,"num_write_kb":12363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"37'335","reported_seq":2431,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939280+0000","last_change":"2024-03-28T15:04:56.745455+0000","last_active":"2024-03-28T15:10:38.939280+0000","last_peered":"2024-03-28T15:10:38.939280+0000","last_clean":"2024-03-28T15:10:38.939280+0000","last_became_active":"2024-03-28T15:02:12.182402+0000","last_became_peered":"2024-03-28T15:02:12.182402+0000","last_unstale":"2024-03-28T15:10:38.939280+0000","last_undegraded":"2024-03-28T15:10:38.939280+0000","last_fullsized":"2024-03-28T15:10:38.939280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:32:46.642308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030686839999999999,"stat_sum":{"num_bytes":2787093,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":329,"num_write_kb":2901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"36'96","reported_seq":163,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.488638+0000","last_change":"2024-03-28T15:01:31.098172+0000","last_active":"2024-03-28T15:10:01.488638+0000","last_peered":"2024-03-28T15:10:01.488638+0000","last_clean":"2024-03-28T15:10:01.488638+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:10:01.488638+0000","last_undegraded":"2024-03-28T15:10:01.488638+0000","last_fullsized":"2024-03-28T15:10:01.488638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:16.015528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24616,"num_write":94,"num_write_kb":8237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"37'348","reported_seq":3089,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.928031+0000","last_change":"2024-03-28T15:04:56.294429+0000","last_active":"2024-03-28T15:10:38.928031+0000","last_peered":"2024-03-28T15:10:38.928031+0000","last_clean":"2024-03-28T15:10:38.928031+0000","last_became_active":"2024-03-28T15:02:12.180011+0000","last_became_peered":"2024-03-28T15:02:12.180011+0000","last_unstale":"2024-03-28T15:10:38.928031+0000","last_undegraded":"2024-03-28T15:10:38.928031+0000","last_fullsized":"2024-03-28T15:10:38.928031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:43.521019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018715680000000001,"stat_sum":{"num_bytes":20266861,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":674,"num_read_kb":16415,"num_write":337,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":51,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.247634+0000","last_change":"2024-03-28T15:01:31.090519+0000","last_active":"2024-03-28T15:09:56.247634+0000","last_peered":"2024-03-28T15:09:56.247634+0000","last_clean":"2024-03-28T15:09:56.247634+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:09:56.247634+0000","last_undegraded":"2024-03-28T15:09:56.247634+0000","last_fullsized":"2024-03-28T15:09:56.247634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"37'358","reported_seq":2567,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.952909+0000","last_change":"2024-03-28T15:04:56.744254+0000","last_active":"2024-03-28T15:10:38.952909+0000","last_peered":"2024-03-28T15:10:38.952909+0000","last_clean":"2024-03-28T15:10:38.952909+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:10:38.952909+0000","last_undegraded":"2024-03-28T15:10:38.952909+0000","last_fullsized":"2024-03-28T15:10:38.952909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:37.876534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022269199999999999,"stat_sum":{"num_bytes":11015620,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":136,"num_read_kb":3039,"num_write":343,"num_write_kb":10934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"37'4","reported_seq":55,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:10.282196+0000","last_change":"2024-03-28T15:01:31.088364+0000","last_active":"2024-03-28T15:10:10.282196+0000","last_peered":"2024-03-28T15:10:10.282196+0000","last_clean":"2024-03-28T15:10:10.282196+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:10:10.282196+0000","last_undegraded":"2024-03-28T15:10:10.282196+0000","last_fullsized":"2024-03-28T15:10:10.282196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"37'344","reported_seq":2987,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.931143+0000","last_change":"2024-03-28T15:04:56.534032+0000","last_active":"2024-03-28T15:10:38.931143+0000","last_peered":"2024-03-28T15:10:38.931143+0000","last_clean":"2024-03-28T15:10:38.931143+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:10:38.931143+0000","last_undegraded":"2024-03-28T15:10:38.931143+0000","last_fullsized":"2024-03-28T15:10:38.931143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:03.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001897143,"stat_sum":{"num_bytes":17657480,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":571,"num_read_kb":16762,"num_write":332,"num_write_kb":17420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"36'220","reported_seq":300,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:23.112150+0000","last_change":"2024-03-28T15:01:31.094837+0000","last_active":"2024-03-28T15:10:23.112150+0000","last_peered":"2024-03-28T15:10:23.112150+0000","last_clean":"2024-03-28T15:10:23.112150+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:10:23.112150+0000","last_undegraded":"2024-03-28T15:10:23.112150+0000","last_fullsized":"2024-03-28T15:10:23.112150+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":83,"num_read_kb":8273,"num_write":219,"num_write_kb":8316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"37'366","reported_seq":2554,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.953014+0000","last_change":"2024-03-28T15:04:56.743437+0000","last_active":"2024-03-28T15:10:38.953014+0000","last_peered":"2024-03-28T15:10:38.953014+0000","last_clean":"2024-03-28T15:10:38.953014+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:10:38.953014+0000","last_undegraded":"2024-03-28T15:10:38.953014+0000","last_fullsized":"2024-03-28T15:10:38.953014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:12:47.009853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001763492,"stat_sum":{"num_bytes":16350020,"num_objects":186,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":115,"num_read_kb":12505,"num_write":360,"num_write_kb":24351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"37'28","reported_seq":88,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:32.560745+0000","last_change":"2024-03-28T15:01:31.092455+0000","last_active":"2024-03-28T15:10:32.560745+0000","last_peered":"2024-03-28T15:10:32.560745+0000","last_clean":"2024-03-28T15:10:32.560745+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:10:32.560745+0000","last_undegraded":"2024-03-28T15:10:32.560745+0000","last_fullsized":"2024-03-28T15:10:32.560745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:04.636216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":29,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"37'290","reported_seq":2751,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.954983+0000","last_change":"2024-03-28T15:04:56.745120+0000","last_active":"2024-03-28T15:10:38.954983+0000","last_peered":"2024-03-28T15:10:38.954983+0000","last_clean":"2024-03-28T15:10:38.954983+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:10:38.954983+0000","last_undegraded":"2024-03-28T15:10:38.954983+0000","last_fullsized":"2024-03-28T15:10:38.954983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:28:19.525504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002782082,"stat_sum":{"num_bytes":11071386,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":387,"num_read_kb":12458,"num_write":285,"num_write_kb":15052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"26'28","reported_seq":90,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:24.170170+0000","last_change":"2024-03-28T15:01:31.092175+0000","last_active":"2024-03-28T15:10:24.170170+0000","last_peered":"2024-03-28T15:10:24.170170+0000","last_clean":"2024-03-28T15:10:24.170170+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:10:24.170170+0000","last_undegraded":"2024-03-28T15:10:24.170170+0000","last_fullsized":"2024-03-28T15:10:24.170170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":28,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"37'337","reported_seq":2700,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957003+0000","last_change":"2024-03-28T15:04:56.745998+0000","last_active":"2024-03-28T15:10:38.957003+0000","last_peered":"2024-03-28T15:10:38.957003+0000","last_clean":"2024-03-28T15:10:38.957003+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:10:38.957003+0000","last_undegraded":"2024-03-28T15:10:38.957003+0000","last_fullsized":"2024-03-28T15:10:38.957003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:36:25.325471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025006389999999998,"stat_sum":{"num_bytes":10266965,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":289,"num_read_kb":13000,"num_write":328,"num_write_kb":14296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"37'120","reported_seq":180,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:34.394134+0000","last_change":"2024-03-28T15:01:31.097901+0000","last_active":"2024-03-28T15:10:34.394134+0000","last_peered":"2024-03-28T15:10:34.394134+0000","last_clean":"2024-03-28T15:10:34.394134+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:10:34.394134+0000","last_undegraded":"2024-03-28T15:10:34.394134+0000","last_fullsized":"2024-03-28T15:10:34.394134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":125,"num_write_kb":4282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"37'362","reported_seq":2665,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957255+0000","last_change":"2024-03-28T15:04:56.200770+0000","last_active":"2024-03-28T15:10:38.957255+0000","last_peered":"2024-03-28T15:10:38.957255+0000","last_clean":"2024-03-28T15:10:38.957255+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:10:38.957255+0000","last_undegraded":"2024-03-28T15:10:38.957255+0000","last_fullsized":"2024-03-28T15:10:38.957255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:09.229020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003899156,"stat_sum":{"num_bytes":10797940,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":229,"num_read_kb":5058,"num_write":352,"num_write_kb":10729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"37'10","reported_seq":62,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:34.525786+0000","last_change":"2024-03-28T15:01:31.086634+0000","last_active":"2024-03-28T15:10:34.525786+0000","last_peered":"2024-03-28T15:10:34.525786+0000","last_clean":"2024-03-28T15:10:34.525786+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:10:34.525786+0000","last_undegraded":"2024-03-28T15:10:34.525786+0000","last_fullsized":"2024-03-28T15:10:34.525786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":4136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"37'326","reported_seq":2660,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957715+0000","last_change":"2024-03-28T15:04:56.744447+0000","last_active":"2024-03-28T15:10:38.957715+0000","last_peered":"2024-03-28T15:10:38.957715+0000","last_clean":"2024-03-28T15:10:38.957715+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:10:38.957715+0000","last_undegraded":"2024-03-28T15:10:38.957715+0000","last_fullsized":"2024-03-28T15:10:38.957715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:43.786965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026010120000000002,"stat_sum":{"num_bytes":14545241,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":260,"num_read_kb":2397,"num_write":318,"num_write_kb":18466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":208,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:57.047088+0000","last_change":"2024-03-28T15:01:20.724163+0000","last_active":"2024-03-28T15:09:57.047088+0000","last_peered":"2024-03-28T15:09:57.047088+0000","last_clean":"2024-03-28T15:09:57.047088+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:09:57.047088+0000","last_undegraded":"2024-03-28T15:09:57.047088+0000","last_fullsized":"2024-03-28T15:09:57.047088+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_clean_scrub_stamp":"2024-03-28T15:01:19.548090+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:15.791893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"37'84","reported_seq":137,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:03.245313+0000","last_change":"2024-03-28T15:01:31.085453+0000","last_active":"2024-03-28T15:10:03.245313+0000","last_peered":"2024-03-28T15:10:03.245313+0000","last_clean":"2024-03-28T15:10:03.245313+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:10:03.245313+0000","last_undegraded":"2024-03-28T15:10:03.245313+0000","last_fullsized":"2024-03-28T15:10:03.245313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":85,"num_write_kb":4165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"37'326","reported_seq":2489,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960193+0000","last_change":"2024-03-28T15:04:56.208514+0000","last_active":"2024-03-28T15:10:38.960193+0000","last_peered":"2024-03-28T15:10:38.960193+0000","last_clean":"2024-03-28T15:10:38.960193+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:10:38.960193+0000","last_undegraded":"2024-03-28T15:10:38.960193+0000","last_fullsized":"2024-03-28T15:10:38.960193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:09.229020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002201228,"stat_sum":{"num_bytes":6136325,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":89,"num_read_kb":4109,"num_write":317,"num_write_kb":10222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"37'225","reported_seq":285,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:30.368284+0000","last_change":"2024-03-28T15:01:31.095439+0000","last_active":"2024-03-28T15:10:30.368284+0000","last_peered":"2024-03-28T15:10:30.368284+0000","last_clean":"2024-03-28T15:10:30.368284+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:10:30.368284+0000","last_undegraded":"2024-03-28T15:10:30.368284+0000","last_fullsized":"2024-03-28T15:10:30.368284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":10,"num_read_kb":28674,"num_write":226,"num_write_kb":12578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"37'330","reported_seq":2633,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932587+0000","last_change":"2024-03-28T15:04:56.198312+0000","last_active":"2024-03-28T15:10:38.932587+0000","last_peered":"2024-03-28T15:10:38.932587+0000","last_clean":"2024-03-28T15:10:38.932587+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:10:38.932587+0000","last_undegraded":"2024-03-28T15:10:38.932587+0000","last_fullsized":"2024-03-28T15:10:38.932587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:30.445328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026072840000000001,"stat_sum":{"num_bytes":10172814,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":229,"num_read_kb":5026,"num_write":323,"num_write_kb":10109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'80","reported_seq":209,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:10.281992+0000","last_change":"2024-03-28T15:01:31.093265+0000","last_active":"2024-03-28T15:10:10.281992+0000","last_peered":"2024-03-28T15:10:10.281992+0000","last_clean":"2024-03-28T15:10:10.281992+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:10:10.281992+0000","last_undegraded":"2024-03-28T15:10:10.281992+0000","last_fullsized":"2024-03-28T15:10:10.281992+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534189,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":569,"num_write":78,"num_write_kb":592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"37'335","reported_seq":2426,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937905+0000","last_change":"2024-03-28T15:04:56.293657+0000","last_active":"2024-03-28T15:10:38.937905+0000","last_peered":"2024-03-28T15:10:38.937905+0000","last_clean":"2024-03-28T15:10:38.937905+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:10:38.937905+0000","last_undegraded":"2024-03-28T15:10:38.937905+0000","last_fullsized":"2024-03-28T15:10:38.937905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:56:09.141610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017724730000000001,"stat_sum":{"num_bytes":1382309,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":326,"num_write_kb":1524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"25'82","reported_seq":140,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.264455+0000","last_change":"2024-03-28T15:01:31.094750+0000","last_active":"2024-03-28T15:10:01.264455+0000","last_peered":"2024-03-28T15:10:01.264455+0000","last_clean":"2024-03-28T15:10:01.264455+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:10:01.264455+0000","last_undegraded":"2024-03-28T15:10:01.264455+0000","last_fullsized":"2024-03-28T15:10:01.264455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":80,"num_write_kb":8230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"37'356","reported_seq":2673,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.933205+0000","last_change":"2024-03-28T15:04:56.198147+0000","last_active":"2024-03-28T15:10:38.933205+0000","last_peered":"2024-03-28T15:10:38.933205+0000","last_clean":"2024-03-28T15:10:38.933205+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:10:38.933205+0000","last_undegraded":"2024-03-28T15:10:38.933205+0000","last_fullsized":"2024-03-28T15:10:38.933205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:59.918813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075601100000000001,"stat_sum":{"num_bytes":5557804,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":243,"num_read_kb":4111,"num_write":349,"num_write_kb":5610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"36'123","reported_seq":177,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.244288+0000","last_change":"2024-03-28T15:01:31.093706+0000","last_active":"2024-03-28T15:10:01.244288+0000","last_peered":"2024-03-28T15:10:01.244288+0000","last_clean":"2024-03-28T15:10:01.244288+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:10:01.244288+0000","last_undegraded":"2024-03-28T15:10:01.244288+0000","last_fullsized":"2024-03-28T15:10:01.244288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4096,"num_write":123,"num_write_kb":4157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"37'346","reported_seq":2940,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956753+0000","last_change":"2024-03-28T15:04:56.200588+0000","last_active":"2024-03-28T15:10:38.956753+0000","last_peered":"2024-03-28T15:10:38.956753+0000","last_clean":"2024-03-28T15:10:38.956753+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:10:38.956753+0000","last_undegraded":"2024-03-28T15:10:38.956753+0000","last_fullsized":"2024-03-28T15:10:38.956753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:08.686987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038365610000000001,"stat_sum":{"num_bytes":18167101,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":520,"num_read_kb":17729,"num_write":341,"num_write_kb":26114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"37'1696","reported_seq":1753,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:23.448944+0000","last_change":"2024-03-28T15:01:31.095126+0000","last_active":"2024-03-28T15:10:23.448944+0000","last_peered":"2024-03-28T15:10:23.448944+0000","last_clean":"2024-03-28T15:10:23.448944+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:10:23.448944+0000","last_undegraded":"2024-03-28T15:10:23.448944+0000","last_fullsized":"2024-03-28T15:10:23.448944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1696,"log_dups_size":0,"ondisk_log_size":1696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:21.669535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":12343,"num_write":1697,"num_write_kb":9020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"37'339","reported_seq":2633,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.930142+0000","last_change":"2024-03-28T15:04:56.293414+0000","last_active":"2024-03-28T15:10:38.930142+0000","last_peered":"2024-03-28T15:10:38.930142+0000","last_clean":"2024-03-28T15:10:38.930142+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:10:38.930142+0000","last_undegraded":"2024-03-28T15:10:38.930142+0000","last_fullsized":"2024-03-28T15:10:38.930142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:05:58.503942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017181130000000001,"stat_sum":{"num_bytes":6397362,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":220,"num_read_kb":4361,"num_write":330,"num_write_kb":6420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'146","reported_seq":206,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.479981+0000","last_change":"2024-03-28T15:01:31.093197+0000","last_active":"2024-03-28T15:10:38.479981+0000","last_peered":"2024-03-28T15:10:38.479981+0000","last_clean":"2024-03-28T15:10:38.479981+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:10:38.479981+0000","last_undegraded":"2024-03-28T15:10:38.479981+0000","last_fullsized":"2024-03-28T15:10:38.479981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":239,"num_write":363,"num_write_kb":1821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"37'324","reported_seq":3170,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.946211+0000","last_change":"2024-03-28T15:04:56.208109+0000","last_active":"2024-03-28T15:10:38.946211+0000","last_peered":"2024-03-28T15:10:38.946211+0000","last_clean":"2024-03-28T15:10:38.946211+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:10:38.946211+0000","last_undegraded":"2024-03-28T15:10:38.946211+0000","last_fullsized":"2024-03-28T15:10:38.946211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:30.228263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002114524,"stat_sum":{"num_bytes":15625536,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":772,"num_read_kb":10204,"num_write":317,"num_write_kb":19524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"36'175","reported_seq":235,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:14.520739+0000","last_change":"2024-03-28T15:01:31.088204+0000","last_active":"2024-03-28T15:10:14.520739+0000","last_peered":"2024-03-28T15:10:14.520739+0000","last_clean":"2024-03-28T15:10:14.520739+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:10:14.520739+0000","last_undegraded":"2024-03-28T15:10:14.520739+0000","last_fullsized":"2024-03-28T15:10:14.520739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":4117,"num_write":175,"num_write_kb":4205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"37'353","reported_seq":3392,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932088+0000","last_change":"2024-03-28T15:04:56.198120+0000","last_active":"2024-03-28T15:10:38.932088+0000","last_peered":"2024-03-28T15:10:38.932088+0000","last_clean":"2024-03-28T15:10:38.932088+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:10:38.932088+0000","last_undegraded":"2024-03-28T15:10:38.932088+0000","last_fullsized":"2024-03-28T15:10:38.932088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:21.937841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025149030000000002,"stat_sum":{"num_bytes":28522087,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":966,"num_read_kb":19238,"num_write":341,"num_write_kb":28027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'224","reported_seq":279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.644049+0000","last_change":"2024-03-28T15:01:31.095371+0000","last_active":"2024-03-28T15:10:29.644049+0000","last_peered":"2024-03-28T15:10:29.644049+0000","last_clean":"2024-03-28T15:10:29.644049+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:10:29.644049+0000","last_undegraded":"2024-03-28T15:10:29.644049+0000","last_fullsized":"2024-03-28T15:10:29.644049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":230,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"37'351","reported_seq":2841,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:33.541714+0000","last_change":"2024-03-28T15:04:56.196825+0000","last_active":"2024-03-28T15:10:33.541714+0000","last_peered":"2024-03-28T15:10:33.541714+0000","last_clean":"2024-03-28T15:10:33.541714+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:10:33.541714+0000","last_undegraded":"2024-03-28T15:10:33.541714+0000","last_fullsized":"2024-03-28T15:10:33.541714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:43:14.280586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083411100000000001,"stat_sum":{"num_bytes":27575979,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":416,"num_read_kb":10935,"num_write":349,"num_write_kb":27109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"36'18","reported_seq":71,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.860756+0000","last_change":"2024-03-28T15:01:31.094542+0000","last_active":"2024-03-28T15:10:02.860756+0000","last_peered":"2024-03-28T15:10:02.860756+0000","last_clean":"2024-03-28T15:10:02.860756+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:10:02.860756+0000","last_undegraded":"2024-03-28T15:10:02.860756+0000","last_fullsized":"2024-03-28T15:10:02.860756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":18,"num_write_kb":4204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"37'362","reported_seq":3011,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938795+0000","last_change":"2024-03-28T15:04:56.208858+0000","last_active":"2024-03-28T15:10:38.938795+0000","last_peered":"2024-03-28T15:10:38.938795+0000","last_clean":"2024-03-28T15:10:38.938795+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:10:38.938795+0000","last_undegraded":"2024-03-28T15:10:38.938795+0000","last_fullsized":"2024-03-28T15:10:38.938795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:36.687483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015964149999999999,"stat_sum":{"num_bytes":23553308,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":575,"num_read_kb":17168,"num_write":351,"num_write_kb":27278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"37'29","reported_seq":84,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:14.524755+0000","last_change":"2024-03-28T15:01:31.094440+0000","last_active":"2024-03-28T15:10:14.524755+0000","last_peered":"2024-03-28T15:10:14.524755+0000","last_clean":"2024-03-28T15:10:14.524755+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:10:14.524755+0000","last_undegraded":"2024-03-28T15:10:14.524755+0000","last_fullsized":"2024-03-28T15:10:14.524755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":31,"num_write_kb":4146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"37'329","reported_seq":2882,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937387+0000","last_change":"2024-03-28T15:04:56.208307+0000","last_active":"2024-03-28T15:10:38.937387+0000","last_peered":"2024-03-28T15:10:38.937387+0000","last_clean":"2024-03-28T15:10:38.937387+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:10:38.937387+0000","last_undegraded":"2024-03-28T15:10:38.937387+0000","last_fullsized":"2024-03-28T15:10:38.937387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:08.686987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022492050000000002,"stat_sum":{"num_bytes":15497791,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":479,"num_read_kb":3997,"num_write":321,"num_write_kb":15303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"37'214","reported_seq":279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:39.457500+0000","last_change":"2024-03-28T15:01:31.096352+0000","last_active":"2024-03-28T15:10:39.457500+0000","last_peered":"2024-03-28T15:10:39.457500+0000","last_clean":"2024-03-28T15:10:39.457500+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:10:39.457500+0000","last_undegraded":"2024-03-28T15:10:39.457500+0000","last_fullsized":"2024-03-28T15:10:39.457500+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9811737,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":18,"num_read_kb":28682,"num_write":209,"num_write_kb":9709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"37'352","reported_seq":2837,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.930151+0000","last_change":"2024-03-28T15:04:56.534400+0000","last_active":"2024-03-28T15:10:38.930151+0000","last_peered":"2024-03-28T15:10:38.930151+0000","last_clean":"2024-03-28T15:10:38.930151+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:10:38.930151+0000","last_undegraded":"2024-03-28T15:10:38.930151+0000","last_fullsized":"2024-03-28T15:10:38.930151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:43.521019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001719216,"stat_sum":{"num_bytes":11206715,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":411,"num_read_kb":8309,"num_write":347,"num_write_kb":11133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"37'327","reported_seq":2611,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.955085+0000","last_change":"2024-03-28T15:04:56.200097+0000","last_active":"2024-03-28T15:10:38.955085+0000","last_peered":"2024-03-28T15:10:38.955085+0000","last_clean":"2024-03-28T15:10:38.955085+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:10:38.955085+0000","last_undegraded":"2024-03-28T15:10:38.955085+0000","last_fullsized":"2024-03-28T15:10:38.955085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:17:15.635128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036538370000000001,"stat_sum":{"num_bytes":7862984,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":211,"num_read_kb":4119,"num_write":311,"num_write_kb":11936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"26'24","reported_seq":85,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.901412+0000","last_change":"2024-03-28T15:01:31.094281+0000","last_active":"2024-03-28T15:10:01.901412+0000","last_peered":"2024-03-28T15:10:01.901412+0000","last_clean":"2024-03-28T15:10:01.901412+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:10:01.901412+0000","last_undegraded":"2024-03-28T15:10:01.901412+0000","last_fullsized":"2024-03-28T15:10:01.901412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":24,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"37'344","reported_seq":3680,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956194+0000","last_change":"2024-03-28T15:04:56.200342+0000","last_active":"2024-03-28T15:10:38.956194+0000","last_peered":"2024-03-28T15:10:38.956194+0000","last_clean":"2024-03-28T15:10:38.956194+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:10:38.956194+0000","last_undegraded":"2024-03-28T15:10:38.956194+0000","last_fullsized":"2024-03-28T15:10:38.956194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:51:56.779182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037913880000000001,"stat_sum":{"num_bytes":22932535,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":1263,"num_read_kb":23613,"num_write":335,"num_write_kb":26620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"37'170","reported_seq":237,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.643756+0000","last_change":"2024-03-28T15:01:31.093773+0000","last_active":"2024-03-28T15:10:29.643756+0000","last_peered":"2024-03-28T15:10:29.643756+0000","last_clean":"2024-03-28T15:10:29.643756+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:10:29.643756+0000","last_undegraded":"2024-03-28T15:10:29.643756+0000","last_fullsized":"2024-03-28T15:10:29.643756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:54.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":36885,"num_write":163,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"37'319","reported_seq":2904,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.943319+0000","last_change":"2024-03-28T15:04:56.197860+0000","last_active":"2024-03-28T15:10:38.943319+0000","last_peered":"2024-03-28T15:10:38.943319+0000","last_clean":"2024-03-28T15:10:38.943319+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:10:38.943319+0000","last_undegraded":"2024-03-28T15:10:38.943319+0000","last_fullsized":"2024-03-28T15:10:38.943319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:22.085679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023175320000000002,"stat_sum":{"num_bytes":14729150,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":511,"num_read_kb":13471,"num_write":311,"num_write_kb":14544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"37'1099","reported_seq":1154,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.257121+0000","last_change":"2024-03-28T15:01:31.096725+0000","last_active":"2024-03-28T15:10:01.257121+0000","last_peered":"2024-03-28T15:10:01.257121+0000","last_clean":"2024-03-28T15:10:01.257121+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:10:01.257121+0000","last_undegraded":"2024-03-28T15:10:01.257121+0000","last_fullsized":"2024-03-28T15:10:01.257121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":1093,"num_write_kb":5019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"37'346","reported_seq":2920,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935556+0000","last_change":"2024-03-28T15:04:56.207406+0000","last_active":"2024-03-28T15:10:38.935556+0000","last_peered":"2024-03-28T15:10:38.935556+0000","last_clean":"2024-03-28T15:10:38.935556+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:10:38.935556+0000","last_undegraded":"2024-03-28T15:10:38.935556+0000","last_fullsized":"2024-03-28T15:10:38.935556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:57:18.708712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015914119999999999,"stat_sum":{"num_bytes":14776390,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":500,"num_read_kb":7280,"num_write":341,"num_write_kb":14609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"36'104","reported_seq":172,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:03.114633+0000","last_change":"2024-03-28T15:01:31.095351+0000","last_active":"2024-03-28T15:10:03.114633+0000","last_peered":"2024-03-28T15:10:03.114633+0000","last_clean":"2024-03-28T15:10:03.114633+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:10:03.114633+0000","last_undegraded":"2024-03-28T15:10:03.114633+0000","last_fullsized":"2024-03-28T15:10:03.114633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":20500,"num_write":104,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":940897971,"num_objects":10589,"num_object_clones":0,"num_object_copies":21178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10589,"num_whiteouts":0,"num_read":30680,"num_read_kb":666406,"num_write":21173,"num_write_kb":1063512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1935458304,"data_stored":1880255846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21704,"ondisk_log_size":21704,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":182313860,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":492,"num_read_kb":424818,"num_write":6917,"num_write_kb":184630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364691456,"data_stored":364627720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6716,"ondisk_log_size":6716,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574965,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425508,"kb_used_data":336092,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":93946332,"statfs":{"total":96636764160,"available":96201043968,"internally_reserved":0,"allocated":344158208,"data_stored":335751183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":91552202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574966,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294192,"kb_used_data":240808,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94077648,"statfs":{"total":96636764160,"available":96335511552,"internally_reserved":0,"allocated":246587392,"data_stored":241270326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574966,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354032,"kb_used_data":308968,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94017808,"statfs":{"total":96636764160,"available":96274235392,"internally_reserved":0,"allocated":316383232,"data_stored":308090025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574966,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":387780,"kb_used_data":338556,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93984060,"statfs":{"total":96636764160,"available":96239677440,"internally_reserved":0,"allocated":346681344,"data_stored":338235386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574965,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405916,"kb_used_data":360852,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93965924,"statfs":{"total":96636764160,"available":96221106176,"internally_reserved":0,"allocated":369512448,"data_stored":360982077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181040,"kb_used_data":149864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94190800,"statfs":{"total":96636764160,"available":96451379200,"internally_reserved":0,"allocated":153460736,"data_stored":148434971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267092,"kb_used_data":217868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94104748,"statfs":{"total":96636764160,"available":96363261952,"internally_reserved":0,"allocated":223096832,"data_stored":217893056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574965,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368928,"kb_used_data":297560,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94002912,"statfs":{"total":96636764160,"available":96258981888,"internally_reserved":0,"allocated":304701440,"data_stored":296790976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50331808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21508096,"data_stored":21505709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":34099200,"data_stored":34089293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51769344,"data_stored":51754903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50332320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":72732672,"data_stored":72726879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261738496,"data_stored":254075801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172343296,"data_stored":167380119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131538944,"data_stored":126748133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334999552,"data_stored":326711655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":294531072,"data_stored":286332021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":274022400,"data_stored":265965766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":195833856,"data_stored":190756877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":270450688,"data_stored":262285474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:10:45.696 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-28T15:10:45.696 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:10:45.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.831+0000 7f3bd75bc640 1 Processor -- start 2024-03-28T15:10:45.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.832+0000 7f3bd75bc640 1 -- start start 2024-03-28T15:10:45.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.832+0000 7f3bd75bc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bd0154bd0 0x7f3bd015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.832+0000 7f3bd75bc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3bd00c57e0 0x7f3bd00cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.832+0000 7f3bd75bc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bd00c6e00 con 0x7f3bd00c53c0 2024-03-28T15:10:45.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.832+0000 7f3bd75bc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f3bd00c5ca0 con 0x7f3bd00c57e0 2024-03-28T15:10:45.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.832+0000 7f3bd75bc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd015b080 con 0x7f3bd0154bd0 2024-03-28T15:10:45.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.832+0000 7f3bd5b59640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3bd00c57e0 0x7f3bd00cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.832+0000 7f3bd4b57640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f3bd00c53c0 0x7f3bd00c66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:52738/0 (socket says 172.21.15.186:52738) 2024-03-28T15:10:45.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.832+0000 7f3bd4b57640 1 -- 172.21.15.186:0/2854656061 learned_addr learned my addr 172.21.15.186:0/2854656061 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:45.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.832+0000 7f3bd5358640 1 --2- 172.21.15.186:0/2854656061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bd0154bd0 0x7f3bd015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.833+0000 7f3bd5b59640 1 -- 172.21.15.186:0/2854656061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bd0154bd0 msgr2=0x7f3bd015a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:45.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.833+0000 7f3bd5b59640 1 --2- 172.21.15.186:0/2854656061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bd0154bd0 0x7f3bd015a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.833+0000 7f3bd5b59640 1 -- 172.21.15.186:0/2854656061 >> v1:172.21.15.186:6789/0 conn(0x7f3bd00c53c0 legacy=0x7f3bd00c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.833+0000 7f3bc7fff640 1 -- 172.21.15.186:0/2854656061 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1760816662 0 0) 0x7f3bd00c6e00 con 0x7f3bd00c53c0 2024-03-28T15:10:45.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.833+0000 7f3bd5b59640 1 -- 172.21.15.186:0/2854656061 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd015b360 con 0x7f3bd00c57e0 2024-03-28T15:10:45.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.833+0000 7f3bd5b59640 1 --2- 172.21.15.186:0/2854656061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd00c57e0 0x7f3bd00cb040 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80097a0 tx=0x7f3bc802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40ddbcf437fd06d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:45.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.834+0000 7f3bc7fff640 1 -- 172.21.15.186:0/2854656061 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8034070 con 0x7f3bd00c57e0 2024-03-28T15:10:45.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.834+0000 7f3bc7fff640 1 -- 172.21.15.186:0/2854656061 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc8030610 con 0x7f3bd00c57e0 2024-03-28T15:10:45.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.834+0000 7f3bc7fff640 1 -- 172.21.15.186:0/2854656061 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc80309e0 con 0x7f3bd00c57e0 2024-03-28T15:10:45.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.834+0000 7f3bd75bc640 1 -- 172.21.15.186:0/2854656061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd00c57e0 msgr2=0x7f3bd00cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.834+0000 7f3bd75bc640 1 --2- 172.21.15.186:0/2854656061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd00c57e0 0x7f3bd00cb040 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80097a0 tx=0x7f3bc802b960 comp rx=0 tx=0).stop 2024-03-28T15:10:45.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.834+0000 7f3bd75bc640 1 -- 172.21.15.186:0/2854656061 shutdown_connections 2024-03-28T15:10:45.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.834+0000 7f3bd75bc640 1 --2- 172.21.15.186:0/2854656061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd00c57e0 0x7f3bd00cb040 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.834+0000 7f3bd75bc640 1 --2- 172.21.15.186:0/2854656061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bd0154bd0 0x7f3bd015a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.834+0000 7f3bd75bc640 1 -- 172.21.15.186:0/2854656061 >> 172.21.15.186:0/2854656061 conn(0x7f3bd00731f0 msgr2=0x7f3bd0073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:45.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.834+0000 7f3bd75bc640 1 -- 172.21.15.186:0/2854656061 shutdown_connections 2024-03-28T15:10:45.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.834+0000 7f3bd75bc640 1 -- 172.21.15.186:0/2854656061 wait complete. 2024-03-28T15:10:45.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.835+0000 7f3bd75bc640 1 Processor -- start 2024-03-28T15:10:45.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.835+0000 7f3bd75bc640 1 -- start start 2024-03-28T15:10:45.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.836+0000 7f3bd75bc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bd00c53c0 0x7f3bd015c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.836+0000 7f3bd75bc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd00c57e0 0x7f3bd014ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.836+0000 7f3bd75bc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bd0154bd0 0x7f3bd0153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.836+0000 7f3bd75bc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd00c61d0 con 0x7f3bd00c57e0 2024-03-28T15:10:45.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.836+0000 7f3bd75bc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd00c5ef0 con 0x7f3bd00c53c0 2024-03-28T15:10:45.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.836+0000 7f3bd75bc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd00cc390 con 0x7f3bd0154bd0 2024-03-28T15:10:45.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.836+0000 7f3bd4b57640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd00c57e0 0x7f3bd014ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.836+0000 7f3bd5358640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bd00c53c0 0x7f3bd015c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.836+0000 7f3bd5b59640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bd0154bd0 0x7f3bd0153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.836+0000 7f3bd4b57640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd00c57e0 0x7f3bd014ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36672/0 (socket says 172.21.15.186:36672) 2024-03-28T15:10:45.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.836+0000 7f3bd4b57640 1 -- 172.21.15.186:0/990190670 learned_addr learned my addr 172.21.15.186:0/990190670 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:45.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bd5358640 1 -- 172.21.15.186:0/990190670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bd0154bd0 msgr2=0x7f3bd0153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bd5358640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bd0154bd0 0x7f3bd0153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bd5358640 1 -- 172.21.15.186:0/990190670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd00c57e0 msgr2=0x7f3bd014ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:45.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bd5358640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd00c57e0 0x7f3bd014ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:45.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bd5358640 1 -- 172.21.15.186:0/990190670 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd01523f0 con 0x7f3bd00c53c0 2024-03-28T15:10:45.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bd4b57640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd00c57e0 0x7f3bd014ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:45.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bd5358640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bd00c53c0 0x7f3bd015c870 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc00ebc0 tx=0x7f3bcc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:45.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bc5ffb640 1 -- 172.21.15.186:0/990190670 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc018020 con 0x7f3bd00c53c0 2024-03-28T15:10:45.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bc5ffb640 1 -- 172.21.15.186:0/990190670 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bcc00cda0 con 0x7f3bd00c53c0 2024-03-28T15:10:45.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bd75bc640 1 -- 172.21.15.186:0/990190670 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd0152660 con 0x7f3bd00c53c0 2024-03-28T15:10:45.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bc5ffb640 1 -- 172.21.15.186:0/990190670 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc0146c0 con 0x7f3bd00c53c0 2024-03-28T15:10:45.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.837+0000 7f3bd75bc640 1 -- 172.21.15.186:0/990190670 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd0152c20 con 0x7f3bd00c53c0 2024-03-28T15:10:45.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.838+0000 7f3bd75bc640 1 -- 172.21.15.186:0/990190670 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bd0069500 con 0x7f3bd00c53c0 2024-03-28T15:10:45.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.839+0000 7f3bc5ffb640 1 -- 172.21.15.186:0/990190670 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3bcc005d10 con 0x7f3bd00c53c0 2024-03-28T15:10:45.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.839+0000 7f3bc5ffb640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3bb406c560 0x7f3bb406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:45.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.839+0000 7f3bc5ffb640 1 -- 172.21.15.186:0/990190670 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f3bcc08c3e0 con 0x7f3bd00c53c0 2024-03-28T15:10:45.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.839+0000 7f3bd4b57640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3bb406c560 0x7f3bb406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:45.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.839+0000 7f3bd4b57640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3bb406c560 0x7f3bb406ea20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0007b10 tx=0x7f3bc0007a60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:45.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:45.844+0000 7f3bc5ffb640 1 -- 172.21.15.186:0/990190670 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f3bcc022030 con 0x7f3bd00c53c0 2024-03-28T15:10:46.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.014+0000 7f3bd75bc640 1 -- 172.21.15.186:0/990190670 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3bd00ce450 con 0x7f3bb406c560 2024-03-28T15:10:46.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.020+0000 7f3bc5ffb640 1 -- 172.21.15.186:0/990190670 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+256109 (secure 0 0 0) 0x7f3bd00ce450 con 0x7f3bb406c560 2024-03-28T15:10:46.024 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:46.025 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:10:46.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 -- 172.21.15.186:0/990190670 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3bb406c560 msgr2=0x7f3bb406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:46.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3bb406c560 0x7f3bb406ea20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0007b10 tx=0x7f3bc0007a60 comp rx=0 tx=0).stop 2024-03-28T15:10:46.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 -- 172.21.15.186:0/990190670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bd00c53c0 msgr2=0x7f3bd015c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:46.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bd00c53c0 0x7f3bd015c870 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc00ebc0 tx=0x7f3bcc00c490 comp rx=0 tx=0).stop 2024-03-28T15:10:46.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 -- 172.21.15.186:0/990190670 shutdown_connections 2024-03-28T15:10:46.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f3bb406c560 0x7f3bb406ea20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bd0154bd0 0x7f3bd0153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd00c57e0 0x7f3bd014ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 --2- 172.21.15.186:0/990190670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bd00c53c0 0x7f3bd015c870 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 -- 172.21.15.186:0/990190670 >> 172.21.15.186:0/990190670 conn(0x7f3bd00731f0 msgr2=0x7f3bd00cb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:46.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 -- 172.21.15.186:0/990190670 shutdown_connections 2024-03-28T15:10:46.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.024+0000 7f3bd75bc640 1 -- 172.21.15.186:0/990190670 wait complete. 2024-03-28T15:10:46.035 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-03-28T15:10:44.750496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1123802199,"num_objects":10721,"num_object_clones":0,"num_object_copies":21442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10721,"num_whiteouts":0,"num_read":31288,"num_read_kb":1091324,"num_write":28249,"num_write_kb":1250558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28537,"ondisk_log_size":28537,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2684488,"kb_used_data":2250568,"kb_used_omap":12,"kb_used_meta":433843,"kb_avail":752290232,"statfs":{"total":773094113280,"available":770345197568,"internally_reserved":0,"allocated":2304581632,"data_stored":2247448000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":444255793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26311273,"num_objects":1458,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":6,"num_read_kb":908,"num_write":1684,"num_write_kb":27163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003096"},"pg_stats":[{"pgid":"3.3f","version":"37'335","reported_seq":3248,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.950296+0000","last_change":"2024-03-28T15:04:56.745680+0000","last_active":"2024-03-28T15:10:38.950296+0000","last_peered":"2024-03-28T15:10:38.950296+0000","last_clean":"2024-03-28T15:10:38.950296+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:10:38.950296+0000","last_undegraded":"2024-03-28T15:10:38.950296+0000","last_fullsized":"2024-03-28T15:10:38.950296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:21:41.971527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024885649999999999,"stat_sum":{"num_bytes":26509954,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":839,"num_read_kb":12103,"num_write":327,"num_write_kb":26054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"37'316","reported_seq":2406,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.930520+0000","last_change":"2024-03-28T15:04:56.294141+0000","last_active":"2024-03-28T15:10:38.930520+0000","last_peered":"2024-03-28T15:10:38.930520+0000","last_clean":"2024-03-28T15:10:38.930520+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:10:38.930520+0000","last_undegraded":"2024-03-28T15:10:38.930520+0000","last_fullsized":"2024-03-28T15:10:38.930520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:43.533807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019334860000000001,"stat_sum":{"num_bytes":1345083,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":314,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"37'329","reported_seq":3088,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.922289+0000","last_change":"2024-03-28T15:04:56.198079+0000","last_active":"2024-03-28T15:10:38.922289+0000","last_peered":"2024-03-28T15:10:38.922289+0000","last_clean":"2024-03-28T15:10:38.922289+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:10:38.922289+0000","last_undegraded":"2024-03-28T15:10:38.922289+0000","last_fullsized":"2024-03-28T15:10:38.922289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:00:42.406838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025151280000000002,"stat_sum":{"num_bytes":12619399,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":685,"num_read_kb":17683,"num_write":321,"num_write_kb":20675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"37'314","reported_seq":3077,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938235+0000","last_change":"2024-03-28T15:04:56.747212+0000","last_active":"2024-03-28T15:10:38.938235+0000","last_peered":"2024-03-28T15:10:38.938235+0000","last_clean":"2024-03-28T15:10:38.938235+0000","last_became_active":"2024-03-28T15:02:12.485356+0000","last_became_peered":"2024-03-28T15:02:12.485356+0000","last_unstale":"2024-03-28T15:10:38.938235+0000","last_undegraded":"2024-03-28T15:10:38.938235+0000","last_fullsized":"2024-03-28T15:10:38.938235+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:51:41.708849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014203849999999999,"stat_sum":{"num_bytes":19355212,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":695,"num_read_kb":14278,"num_write":306,"num_write_kb":19012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"37'363","reported_seq":2551,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938805+0000","last_change":"2024-03-28T15:04:56.745133+0000","last_active":"2024-03-28T15:10:38.938805+0000","last_peered":"2024-03-28T15:10:38.938805+0000","last_clean":"2024-03-28T15:10:38.938805+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:10:38.938805+0000","last_undegraded":"2024-03-28T15:10:38.938805+0000","last_fullsized":"2024-03-28T15:10:38.938805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:21.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002796064,"stat_sum":{"num_bytes":10679980,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":115,"num_read_kb":4114,"num_write":361,"num_write_kb":10554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"37'327","reported_seq":2639,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.946373+0000","last_change":"2024-03-28T15:04:56.200817+0000","last_active":"2024-03-28T15:10:38.946373+0000","last_peered":"2024-03-28T15:10:38.946373+0000","last_clean":"2024-03-28T15:10:38.946373+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:10:38.946373+0000","last_undegraded":"2024-03-28T15:10:38.946373+0000","last_fullsized":"2024-03-28T15:10:38.946373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:03:09.771351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003798718,"stat_sum":{"num_bytes":10128543,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":239,"num_read_kb":4123,"num_write":321,"num_write_kb":10056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"37'354","reported_seq":3455,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.950756+0000","last_change":"2024-03-28T15:04:56.743147+0000","last_active":"2024-03-28T15:10:38.950756+0000","last_peered":"2024-03-28T15:10:38.950756+0000","last_clean":"2024-03-28T15:10:38.950756+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:10:38.950756+0000","last_undegraded":"2024-03-28T15:10:38.950756+0000","last_fullsized":"2024-03-28T15:10:38.950756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:20.151357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001484766,"stat_sum":{"num_bytes":23645872,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":1028,"num_read_kb":17521,"num_write":350,"num_write_kb":23210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"37'334","reported_seq":2944,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.959962+0000","last_change":"2024-03-28T15:04:56.208811+0000","last_active":"2024-03-28T15:10:38.959962+0000","last_peered":"2024-03-28T15:10:38.959962+0000","last_clean":"2024-03-28T15:10:38.959962+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:10:38.959962+0000","last_undegraded":"2024-03-28T15:10:38.959962+0000","last_fullsized":"2024-03-28T15:10:38.959962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:57.578217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017148910000000001,"stat_sum":{"num_bytes":19312172,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":537,"num_read_kb":7179,"num_write":326,"num_write_kb":18943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"37'330","reported_seq":2712,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960654+0000","last_change":"2024-03-28T15:04:56.745714+0000","last_active":"2024-03-28T15:10:38.960654+0000","last_peered":"2024-03-28T15:10:38.960654+0000","last_clean":"2024-03-28T15:10:38.960654+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:10:38.960654+0000","last_undegraded":"2024-03-28T15:10:38.960654+0000","last_fullsized":"2024-03-28T15:10:38.960654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:27:32.222447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023275240000000001,"stat_sum":{"num_bytes":6398085,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":309,"num_read_kb":8863,"num_write":320,"num_write_kb":10489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"37'327","reported_seq":3210,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.961682+0000","last_change":"2024-03-28T15:04:56.534451+0000","last_active":"2024-03-28T15:10:38.961682+0000","last_peered":"2024-03-28T15:10:38.961682+0000","last_clean":"2024-03-28T15:10:38.961682+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:10:38.961682+0000","last_undegraded":"2024-03-28T15:10:38.961682+0000","last_fullsized":"2024-03-28T15:10:38.961682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:43.533807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019348060000000001,"stat_sum":{"num_bytes":19157707,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":810,"num_read_kb":17508,"num_write":311,"num_write_kb":18862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"37'303","reported_seq":3011,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.959082+0000","last_change":"2024-03-28T15:04:56.534170+0000","last_active":"2024-03-28T15:10:38.959082+0000","last_peered":"2024-03-28T15:10:38.959082+0000","last_clean":"2024-03-28T15:10:38.959082+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:10:38.959082+0000","last_undegraded":"2024-03-28T15:10:38.959082+0000","last_fullsized":"2024-03-28T15:10:38.959082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:20.495757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018723170000000001,"stat_sum":{"num_bytes":18249049,"num_objects":129,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":635,"num_read_kb":13018,"num_write":297,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"37'296","reported_seq":2603,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.934653+0000","last_change":"2024-03-28T15:04:56.199593+0000","last_active":"2024-03-28T15:10:38.934653+0000","last_peered":"2024-03-28T15:10:38.934653+0000","last_clean":"2024-03-28T15:10:38.934653+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:10:38.934653+0000","last_undegraded":"2024-03-28T15:10:38.934653+0000","last_fullsized":"2024-03-28T15:10:38.934653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:50.527037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013925400000000001,"stat_sum":{"num_bytes":6129603,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":234,"num_read_kb":4757,"num_write":288,"num_write_kb":10236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"37'297","reported_seq":2859,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936011+0000","last_change":"2024-03-28T15:04:56.207750+0000","last_active":"2024-03-28T15:10:38.936011+0000","last_peered":"2024-03-28T15:10:38.936011+0000","last_clean":"2024-03-28T15:10:38.936011+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:10:38.936011+0000","last_undegraded":"2024-03-28T15:10:38.936011+0000","last_fullsized":"2024-03-28T15:10:38.936011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:17:15.635128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017572180000000001,"stat_sum":{"num_bytes":15029757,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":489,"num_read_kb":12548,"num_write":286,"num_write_kb":23016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"37'341","reported_seq":2613,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932033+0000","last_change":"2024-03-28T15:04:56.197644+0000","last_active":"2024-03-28T15:10:38.932033+0000","last_peered":"2024-03-28T15:10:38.932033+0000","last_clean":"2024-03-28T15:10:38.932033+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:10:38.932033+0000","last_undegraded":"2024-03-28T15:10:38.932033+0000","last_fullsized":"2024-03-28T15:10:38.932033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:08.587017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002253971,"stat_sum":{"num_bytes":6664925,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":199,"num_read_kb":4930,"num_write":335,"num_write_kb":6638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"37'357","reported_seq":3528,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.926788+0000","last_change":"2024-03-28T15:04:56.293889+0000","last_active":"2024-03-28T15:10:38.926788+0000","last_peered":"2024-03-28T15:10:38.926788+0000","last_clean":"2024-03-28T15:10:38.926788+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:10:38.926788+0000","last_undegraded":"2024-03-28T15:10:38.926788+0000","last_fullsized":"2024-03-28T15:10:38.926788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:20.495757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179673,"stat_sum":{"num_bytes":23044368,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":1098,"num_read_kb":19203,"num_write":350,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"37'357","reported_seq":2912,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.945394+0000","last_change":"2024-03-28T15:04:56.200579+0000","last_active":"2024-03-28T15:10:38.945394+0000","last_peered":"2024-03-28T15:10:38.945394+0000","last_clean":"2024-03-28T15:10:38.945394+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:10:38.945394+0000","last_undegraded":"2024-03-28T15:10:38.945394+0000","last_fullsized":"2024-03-28T15:10:38.945394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:30.228263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039634529999999996,"stat_sum":{"num_bytes":16147473,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":482,"num_read_kb":8871,"num_write":348,"num_write_kb":19203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"37'349","reported_seq":2573,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960198+0000","last_change":"2024-03-28T15:04:56.201087+0000","last_active":"2024-03-28T15:10:38.960198+0000","last_peered":"2024-03-28T15:10:38.960198+0000","last_clean":"2024-03-28T15:10:38.960198+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:10:38.960198+0000","last_undegraded":"2024-03-28T15:10:38.960198+0000","last_fullsized":"2024-03-28T15:10:38.960198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:36.687483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027424319999999999,"stat_sum":{"num_bytes":10498072,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":151,"num_read_kb":9005,"num_write":340,"num_write_kb":14498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"37'311","reported_seq":2414,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936309+0000","last_change":"2024-03-28T15:04:56.746916+0000","last_active":"2024-03-28T15:10:38.936309+0000","last_peered":"2024-03-28T15:10:38.936309+0000","last_clean":"2024-03-28T15:10:38.936309+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:10:38.936309+0000","last_undegraded":"2024-03-28T15:10:38.936309+0000","last_fullsized":"2024-03-28T15:10:38.936309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:20:35.393196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179399,"stat_sum":{"num_bytes":9508598,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":30,"num_read_kb":42,"num_write":300,"num_write_kb":9445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"37'324","reported_seq":2710,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939190+0000","last_change":"2024-03-28T15:04:56.745421+0000","last_active":"2024-03-28T15:10:38.939190+0000","last_peered":"2024-03-28T15:10:38.939190+0000","last_clean":"2024-03-28T15:10:38.939190+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:10:38.939190+0000","last_undegraded":"2024-03-28T15:10:38.939190+0000","last_fullsized":"2024-03-28T15:10:38.939190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:50.382005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030062439999999999,"stat_sum":{"num_bytes":14185939,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":312,"num_read_kb":12813,"num_write":317,"num_write_kb":18119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"37'356","reported_seq":3288,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937246+0000","last_change":"2024-03-28T15:04:56.534664+0000","last_active":"2024-03-28T15:10:38.937246+0000","last_peered":"2024-03-28T15:10:38.937246+0000","last_clean":"2024-03-28T15:10:38.937246+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:10:38.937246+0000","last_undegraded":"2024-03-28T15:10:38.937246+0000","last_fullsized":"2024-03-28T15:10:38.937246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:10.614269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001798584,"stat_sum":{"num_bytes":30036364,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":858,"num_read_kb":15492,"num_write":348,"num_write_kb":37706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"37'342","reported_seq":2724,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957518+0000","last_change":"2024-03-28T15:04:56.198376+0000","last_active":"2024-03-28T15:10:38.957518+0000","last_peered":"2024-03-28T15:10:38.957518+0000","last_clean":"2024-03-28T15:10:38.957518+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:10:38.957518+0000","last_undegraded":"2024-03-28T15:10:38.957518+0000","last_fullsized":"2024-03-28T15:10:38.957518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:55:39.836954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025216930000000002,"stat_sum":{"num_bytes":10421413,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":308,"num_read_kb":8356,"num_write":331,"num_write_kb":10350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"37'337","reported_seq":2523,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935072+0000","last_change":"2024-03-28T15:04:56.746844+0000","last_active":"2024-03-28T15:10:38.935072+0000","last_peered":"2024-03-28T15:10:38.935072+0000","last_clean":"2024-03-28T15:10:38.935072+0000","last_became_active":"2024-03-28T15:02:12.178196+0000","last_became_peered":"2024-03-28T15:02:12.178196+0000","last_unstale":"2024-03-28T15:10:38.935072+0000","last_undegraded":"2024-03-28T15:10:38.935072+0000","last_fullsized":"2024-03-28T15:10:38.935072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:59:11.409442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001195878,"stat_sum":{"num_bytes":9431253,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":119,"num_read_kb":4113,"num_write":327,"num_write_kb":13479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"37'331","reported_seq":2998,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939818+0000","last_change":"2024-03-28T15:04:56.207990+0000","last_active":"2024-03-28T15:10:38.939818+0000","last_peered":"2024-03-28T15:10:38.939818+0000","last_clean":"2024-03-28T15:10:38.939818+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:10:38.939818+0000","last_undegraded":"2024-03-28T15:10:38.939818+0000","last_fullsized":"2024-03-28T15:10:38.939818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:51:56.779182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020004409999999999,"stat_sum":{"num_bytes":16319296,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":593,"num_read_kb":18222,"num_write":323,"num_write_kb":20203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"37'356","reported_seq":2799,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956957+0000","last_change":"2024-03-28T15:04:56.200922+0000","last_active":"2024-03-28T15:10:38.956957+0000","last_peered":"2024-03-28T15:10:38.956957+0000","last_clean":"2024-03-28T15:10:38.956957+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:10:38.956957+0000","last_undegraded":"2024-03-28T15:10:38.956957+0000","last_fullsized":"2024-03-28T15:10:38.956957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:57.578217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031981639999999999,"stat_sum":{"num_bytes":14893759,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":369,"num_read_kb":10097,"num_write":349,"num_write_kb":14727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"37'314","reported_seq":3255,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.931560+0000","last_change":"2024-03-28T15:04:56.207665+0000","last_active":"2024-03-28T15:10:38.931560+0000","last_peered":"2024-03-28T15:10:38.931560+0000","last_clean":"2024-03-28T15:10:38.931560+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:10:38.931560+0000","last_undegraded":"2024-03-28T15:10:38.931560+0000","last_fullsized":"2024-03-28T15:10:38.931560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:36:17.765596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001813119,"stat_sum":{"num_bytes":18699537,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":867,"num_read_kb":15099,"num_write":310,"num_write_kb":18421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"37'377","reported_seq":2618,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960107+0000","last_change":"2024-03-28T15:04:56.534606+0000","last_active":"2024-03-28T15:10:38.960107+0000","last_peered":"2024-03-28T15:10:38.960107+0000","last_clean":"2024-03-28T15:10:38.960107+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:10:38.960107+0000","last_undegraded":"2024-03-28T15:10:38.960107+0000","last_fullsized":"2024-03-28T15:10:38.960107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:23:24.123040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019272180000000001,"stat_sum":{"num_bytes":11206818,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":167,"num_read_kb":4115,"num_write":363,"num_write_kb":11124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"37'320","reported_seq":2923,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936502+0000","last_change":"2024-03-28T15:04:56.294225+0000","last_active":"2024-03-28T15:10:38.936502+0000","last_peered":"2024-03-28T15:10:38.936502+0000","last_clean":"2024-03-28T15:10:38.936502+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:10:38.936502+0000","last_undegraded":"2024-03-28T15:10:38.936502+0000","last_fullsized":"2024-03-28T15:10:38.936502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:23:24.123040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019373070000000001,"stat_sum":{"num_bytes":22270707,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":529,"num_read_kb":12598,"num_write":314,"num_write_kb":21911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"37'315","reported_seq":3227,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.942957+0000","last_change":"2024-03-28T15:04:56.208607+0000","last_active":"2024-03-28T15:10:38.942957+0000","last_peered":"2024-03-28T15:10:38.942957+0000","last_clean":"2024-03-28T15:10:38.942957+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:10:38.942957+0000","last_undegraded":"2024-03-28T15:10:38.942957+0000","last_fullsized":"2024-03-28T15:10:38.942957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:03:09.771351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020980170000000002,"stat_sum":{"num_bytes":17909578,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":838,"num_read_kb":18539,"num_write":311,"num_write_kb":20928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"37'358","reported_seq":3192,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935632+0000","last_change":"2024-03-28T15:04:56.198529+0000","last_active":"2024-03-28T15:10:38.935632+0000","last_peered":"2024-03-28T15:10:38.935632+0000","last_clean":"2024-03-28T15:10:38.935632+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:10:38.935632+0000","last_undegraded":"2024-03-28T15:10:38.935632+0000","last_fullsized":"2024-03-28T15:10:38.935632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:41:01.927661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026139499999999999,"stat_sum":{"num_bytes":26288347,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":760,"num_read_kb":8154,"num_write":350,"num_write_kb":25854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"37'359","reported_seq":2900,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956962+0000","last_change":"2024-03-28T15:04:56.200966+0000","last_active":"2024-03-28T15:10:38.956962+0000","last_peered":"2024-03-28T15:10:38.956962+0000","last_clean":"2024-03-28T15:10:38.956962+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:10:38.956962+0000","last_undegraded":"2024-03-28T15:10:38.956962+0000","last_fullsized":"2024-03-28T15:10:38.956962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:37.204800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036450829999999999,"stat_sum":{"num_bytes":10387544,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":467,"num_read_kb":8853,"num_write":352,"num_write_kb":10266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"37'386","reported_seq":2968,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.942998+0000","last_change":"2024-03-28T15:04:56.747393+0000","last_active":"2024-03-28T15:10:38.942998+0000","last_peered":"2024-03-28T15:10:38.942998+0000","last_clean":"2024-03-28T15:10:38.942998+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:10:38.942998+0000","last_undegraded":"2024-03-28T15:10:38.942998+0000","last_fullsized":"2024-03-28T15:10:38.942998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:39:11.217145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001020962,"stat_sum":{"num_bytes":12939697,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":508,"num_read_kb":8898,"num_write":372,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"37'384","reported_seq":2787,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932311+0000","last_change":"2024-03-28T15:04:56.534204+0000","last_active":"2024-03-28T15:10:38.932311+0000","last_peered":"2024-03-28T15:10:38.932311+0000","last_clean":"2024-03-28T15:10:38.932311+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:10:38.932311+0000","last_undegraded":"2024-03-28T15:10:38.932311+0000","last_fullsized":"2024-03-28T15:10:38.932311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:56:09.141610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019055229999999999,"stat_sum":{"num_bytes":11304826,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":329,"num_read_kb":5287,"num_write":374,"num_write_kb":11236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"37'322","reported_seq":2854,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.921473+0000","last_change":"2024-03-28T15:04:56.196629+0000","last_active":"2024-03-28T15:10:38.921473+0000","last_peered":"2024-03-28T15:10:38.921473+0000","last_clean":"2024-03-28T15:10:38.921473+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:10:38.921473+0000","last_undegraded":"2024-03-28T15:10:38.921473+0000","last_fullsized":"2024-03-28T15:10:38.921473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:51.357070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078031800000000001,"stat_sum":{"num_bytes":14482293,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":459,"num_read_kb":10396,"num_write":315,"num_write_kb":14305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"36'18","reported_seq":74,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.776351+0000","last_change":"2024-03-28T15:01:31.097415+0000","last_active":"2024-03-28T15:09:56.776351+0000","last_peered":"2024-03-28T15:09:56.776351+0000","last_clean":"2024-03-28T15:09:56.776351+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:09:56.776351+0000","last_undegraded":"2024-03-28T15:09:56.776351+0000","last_fullsized":"2024-03-28T15:09:56.776351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":16384,"num_write":21,"num_write_kb":4230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"37'325","reported_seq":2638,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.934802+0000","last_change":"2024-03-28T15:04:56.198560+0000","last_active":"2024-03-28T15:10:38.934802+0000","last_peered":"2024-03-28T15:10:38.934802+0000","last_clean":"2024-03-28T15:10:38.934802+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:10:38.934802+0000","last_undegraded":"2024-03-28T15:10:38.934802+0000","last_fullsized":"2024-03-28T15:10:38.934802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:51.739886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023690579999999998,"stat_sum":{"num_bytes":11043752,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":240,"num_read_kb":4317,"num_write":317,"num_write_kb":10950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"37'647","reported_seq":711,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.472792+0000","last_change":"2024-03-28T15:01:31.088437+0000","last_active":"2024-03-28T15:10:02.472792+0000","last_peered":"2024-03-28T15:10:02.472792+0000","last_clean":"2024-03-28T15:10:02.472792+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:10:02.472792+0000","last_undegraded":"2024-03-28T15:10:02.472792+0000","last_fullsized":"2024-03-28T15:10:02.472792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":34526,"num_write":642,"num_write_kb":12757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"36'58","reported_seq":112,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.775891+0000","last_change":"2024-03-28T15:01:31.097499+0000","last_active":"2024-03-28T15:09:56.775891+0000","last_peered":"2024-03-28T15:09:56.775891+0000","last_clean":"2024-03-28T15:09:56.775891+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:09:56.775891+0000","last_undegraded":"2024-03-28T15:09:56.775891+0000","last_fullsized":"2024-03-28T15:09:56.775891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":62,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"37'402","reported_seq":3693,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936725+0000","last_change":"2024-03-28T15:04:56.208655+0000","last_active":"2024-03-28T15:10:38.936725+0000","last_peered":"2024-03-28T15:10:38.936725+0000","last_clean":"2024-03-28T15:10:38.936725+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:10:38.936725+0000","last_undegraded":"2024-03-28T15:10:38.936725+0000","last_fullsized":"2024-03-28T15:10:38.936725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:37.204800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017492600000000001,"stat_sum":{"num_bytes":22338843,"num_objects":188,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":1218,"num_read_kb":22454,"num_write":394,"num_write_kb":30214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"37'604","reported_seq":661,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:30.364435+0000","last_change":"2024-03-28T15:01:31.086252+0000","last_active":"2024-03-28T15:10:30.364435+0000","last_peered":"2024-03-28T15:10:30.364435+0000","last_clean":"2024-03-28T15:10:30.364435+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:10:30.364435+0000","last_undegraded":"2024-03-28T15:10:30.364435+0000","last_fullsized":"2024-03-28T15:10:30.364435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":603,"num_write_kb":12492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"37'341","reported_seq":3347,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937426+0000","last_change":"2024-03-28T15:04:56.293936+0000","last_active":"2024-03-28T15:10:38.937426+0000","last_peered":"2024-03-28T15:10:38.937426+0000","last_clean":"2024-03-28T15:10:38.937426+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:10:38.937426+0000","last_undegraded":"2024-03-28T15:10:38.937426+0000","last_fullsized":"2024-03-28T15:10:38.937426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:03.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022536280000000001,"stat_sum":{"num_bytes":19647424,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":933,"num_read_kb":22521,"num_write":330,"num_write_kb":23406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"37'5","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.246758+0000","last_change":"2024-03-28T15:01:31.092338+0000","last_active":"2024-03-28T15:09:56.246758+0000","last_peered":"2024-03-28T15:09:56.246758+0000","last_clean":"2024-03-28T15:09:56.246758+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:09:56.246758+0000","last_undegraded":"2024-03-28T15:09:56.246758+0000","last_fullsized":"2024-03-28T15:09:56.246758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"37'366","reported_seq":3228,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960208+0000","last_change":"2024-03-28T15:04:56.200271+0000","last_active":"2024-03-28T15:10:38.960208+0000","last_peered":"2024-03-28T15:10:38.960208+0000","last_clean":"2024-03-28T15:10:38.960208+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:10:38.960208+0000","last_undegraded":"2024-03-28T15:10:38.960208+0000","last_fullsized":"2024-03-28T15:10:38.960208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:36:17.765596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043389099999999996,"stat_sum":{"num_bytes":22872414,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":789,"num_read_kb":19564,"num_write":357,"num_write_kb":22493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"37'70","reported_seq":124,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.349608+0000","last_change":"2024-03-28T15:01:31.088909+0000","last_active":"2024-03-28T15:10:02.349608+0000","last_peered":"2024-03-28T15:10:02.349608+0000","last_clean":"2024-03-28T15:10:02.349608+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:10:02.349608+0000","last_undegraded":"2024-03-28T15:10:02.349608+0000","last_fullsized":"2024-03-28T15:10:02.349608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":66,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"37'323","reported_seq":2969,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.943307+0000","last_change":"2024-03-28T15:04:56.197873+0000","last_active":"2024-03-28T15:10:38.943307+0000","last_peered":"2024-03-28T15:10:38.943307+0000","last_clean":"2024-03-28T15:10:38.943307+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:10:38.943307+0000","last_undegraded":"2024-03-28T15:10:38.943307+0000","last_fullsized":"2024-03-28T15:10:38.943307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:52.072532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023945189999999999,"stat_sum":{"num_bytes":10186807,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":573,"num_read_kb":8919,"num_write":314,"num_write_kb":10118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"36'193","reported_seq":257,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.698311+0000","last_change":"2024-03-28T15:01:31.095935+0000","last_active":"2024-03-28T15:10:02.698311+0000","last_peered":"2024-03-28T15:10:02.698311+0000","last_clean":"2024-03-28T15:10:02.698311+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:10:02.698311+0000","last_undegraded":"2024-03-28T15:10:02.698311+0000","last_fullsized":"2024-03-28T15:10:02.698311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":4122,"num_write":194,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"37'321","reported_seq":3146,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957720+0000","last_change":"2024-03-28T15:04:56.199307+0000","last_active":"2024-03-28T15:10:38.957720+0000","last_peered":"2024-03-28T15:10:38.957720+0000","last_clean":"2024-03-28T15:10:38.957720+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:10:38.957720+0000","last_undegraded":"2024-03-28T15:10:38.957720+0000","last_fullsized":"2024-03-28T15:10:38.957720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:32:03.268809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254104,"stat_sum":{"num_bytes":14752712,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":752,"num_read_kb":16498,"num_write":310,"num_write_kb":18664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"37'122","reported_seq":201,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.389422+0000","last_change":"2024-03-28T15:01:31.089380+0000","last_active":"2024-03-28T15:10:29.389422+0000","last_peered":"2024-03-28T15:10:29.389422+0000","last_clean":"2024-03-28T15:10:29.389422+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:10:29.389422+0000","last_undegraded":"2024-03-28T15:10:29.389422+0000","last_fullsized":"2024-03-28T15:10:29.389422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32804,"num_write":115,"num_write_kb":12363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"37'335","reported_seq":2431,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939280+0000","last_change":"2024-03-28T15:04:56.745455+0000","last_active":"2024-03-28T15:10:38.939280+0000","last_peered":"2024-03-28T15:10:38.939280+0000","last_clean":"2024-03-28T15:10:38.939280+0000","last_became_active":"2024-03-28T15:02:12.182402+0000","last_became_peered":"2024-03-28T15:02:12.182402+0000","last_unstale":"2024-03-28T15:10:38.939280+0000","last_undegraded":"2024-03-28T15:10:38.939280+0000","last_fullsized":"2024-03-28T15:10:38.939280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:32:46.642308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030686839999999999,"stat_sum":{"num_bytes":2787093,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":329,"num_write_kb":2901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"36'96","reported_seq":163,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.488638+0000","last_change":"2024-03-28T15:01:31.098172+0000","last_active":"2024-03-28T15:10:01.488638+0000","last_peered":"2024-03-28T15:10:01.488638+0000","last_clean":"2024-03-28T15:10:01.488638+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:10:01.488638+0000","last_undegraded":"2024-03-28T15:10:01.488638+0000","last_fullsized":"2024-03-28T15:10:01.488638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:16.015528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24616,"num_write":94,"num_write_kb":8237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"37'348","reported_seq":3089,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.928031+0000","last_change":"2024-03-28T15:04:56.294429+0000","last_active":"2024-03-28T15:10:38.928031+0000","last_peered":"2024-03-28T15:10:38.928031+0000","last_clean":"2024-03-28T15:10:38.928031+0000","last_became_active":"2024-03-28T15:02:12.180011+0000","last_became_peered":"2024-03-28T15:02:12.180011+0000","last_unstale":"2024-03-28T15:10:38.928031+0000","last_undegraded":"2024-03-28T15:10:38.928031+0000","last_fullsized":"2024-03-28T15:10:38.928031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:43.521019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018715680000000001,"stat_sum":{"num_bytes":20266861,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":674,"num_read_kb":16415,"num_write":337,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":51,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.247634+0000","last_change":"2024-03-28T15:01:31.090519+0000","last_active":"2024-03-28T15:09:56.247634+0000","last_peered":"2024-03-28T15:09:56.247634+0000","last_clean":"2024-03-28T15:09:56.247634+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:09:56.247634+0000","last_undegraded":"2024-03-28T15:09:56.247634+0000","last_fullsized":"2024-03-28T15:09:56.247634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"37'358","reported_seq":2567,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.952909+0000","last_change":"2024-03-28T15:04:56.744254+0000","last_active":"2024-03-28T15:10:38.952909+0000","last_peered":"2024-03-28T15:10:38.952909+0000","last_clean":"2024-03-28T15:10:38.952909+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:10:38.952909+0000","last_undegraded":"2024-03-28T15:10:38.952909+0000","last_fullsized":"2024-03-28T15:10:38.952909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:37.876534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022269199999999999,"stat_sum":{"num_bytes":11015620,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":136,"num_read_kb":3039,"num_write":343,"num_write_kb":10934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"37'4","reported_seq":55,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:10.282196+0000","last_change":"2024-03-28T15:01:31.088364+0000","last_active":"2024-03-28T15:10:10.282196+0000","last_peered":"2024-03-28T15:10:10.282196+0000","last_clean":"2024-03-28T15:10:10.282196+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:10:10.282196+0000","last_undegraded":"2024-03-28T15:10:10.282196+0000","last_fullsized":"2024-03-28T15:10:10.282196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"37'344","reported_seq":2987,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.931143+0000","last_change":"2024-03-28T15:04:56.534032+0000","last_active":"2024-03-28T15:10:38.931143+0000","last_peered":"2024-03-28T15:10:38.931143+0000","last_clean":"2024-03-28T15:10:38.931143+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:10:38.931143+0000","last_undegraded":"2024-03-28T15:10:38.931143+0000","last_fullsized":"2024-03-28T15:10:38.931143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:03.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001897143,"stat_sum":{"num_bytes":17657480,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":571,"num_read_kb":16762,"num_write":332,"num_write_kb":17420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"36'220","reported_seq":300,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:23.112150+0000","last_change":"2024-03-28T15:01:31.094837+0000","last_active":"2024-03-28T15:10:23.112150+0000","last_peered":"2024-03-28T15:10:23.112150+0000","last_clean":"2024-03-28T15:10:23.112150+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:10:23.112150+0000","last_undegraded":"2024-03-28T15:10:23.112150+0000","last_fullsized":"2024-03-28T15:10:23.112150+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":83,"num_read_kb":8273,"num_write":219,"num_write_kb":8316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"37'366","reported_seq":2554,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.953014+0000","last_change":"2024-03-28T15:04:56.743437+0000","last_active":"2024-03-28T15:10:38.953014+0000","last_peered":"2024-03-28T15:10:38.953014+0000","last_clean":"2024-03-28T15:10:38.953014+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:10:38.953014+0000","last_undegraded":"2024-03-28T15:10:38.953014+0000","last_fullsized":"2024-03-28T15:10:38.953014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:12:47.009853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001763492,"stat_sum":{"num_bytes":16350020,"num_objects":186,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":115,"num_read_kb":12505,"num_write":360,"num_write_kb":24351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"37'28","reported_seq":88,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:32.560745+0000","last_change":"2024-03-28T15:01:31.092455+0000","last_active":"2024-03-28T15:10:32.560745+0000","last_peered":"2024-03-28T15:10:32.560745+0000","last_clean":"2024-03-28T15:10:32.560745+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:10:32.560745+0000","last_undegraded":"2024-03-28T15:10:32.560745+0000","last_fullsized":"2024-03-28T15:10:32.560745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:04.636216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":29,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"37'290","reported_seq":2751,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.954983+0000","last_change":"2024-03-28T15:04:56.745120+0000","last_active":"2024-03-28T15:10:38.954983+0000","last_peered":"2024-03-28T15:10:38.954983+0000","last_clean":"2024-03-28T15:10:38.954983+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:10:38.954983+0000","last_undegraded":"2024-03-28T15:10:38.954983+0000","last_fullsized":"2024-03-28T15:10:38.954983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:28:19.525504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002782082,"stat_sum":{"num_bytes":11071386,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":387,"num_read_kb":12458,"num_write":285,"num_write_kb":15052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"26'28","reported_seq":90,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:24.170170+0000","last_change":"2024-03-28T15:01:31.092175+0000","last_active":"2024-03-28T15:10:24.170170+0000","last_peered":"2024-03-28T15:10:24.170170+0000","last_clean":"2024-03-28T15:10:24.170170+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:10:24.170170+0000","last_undegraded":"2024-03-28T15:10:24.170170+0000","last_fullsized":"2024-03-28T15:10:24.170170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":28,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"37'337","reported_seq":2700,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957003+0000","last_change":"2024-03-28T15:04:56.745998+0000","last_active":"2024-03-28T15:10:38.957003+0000","last_peered":"2024-03-28T15:10:38.957003+0000","last_clean":"2024-03-28T15:10:38.957003+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:10:38.957003+0000","last_undegraded":"2024-03-28T15:10:38.957003+0000","last_fullsized":"2024-03-28T15:10:38.957003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:36:25.325471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025006389999999998,"stat_sum":{"num_bytes":10266965,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":289,"num_read_kb":13000,"num_write":328,"num_write_kb":14296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"37'120","reported_seq":180,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:34.394134+0000","last_change":"2024-03-28T15:01:31.097901+0000","last_active":"2024-03-28T15:10:34.394134+0000","last_peered":"2024-03-28T15:10:34.394134+0000","last_clean":"2024-03-28T15:10:34.394134+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:10:34.394134+0000","last_undegraded":"2024-03-28T15:10:34.394134+0000","last_fullsized":"2024-03-28T15:10:34.394134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":125,"num_write_kb":4282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"37'362","reported_seq":2665,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957255+0000","last_change":"2024-03-28T15:04:56.200770+0000","last_active":"2024-03-28T15:10:38.957255+0000","last_peered":"2024-03-28T15:10:38.957255+0000","last_clean":"2024-03-28T15:10:38.957255+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:10:38.957255+0000","last_undegraded":"2024-03-28T15:10:38.957255+0000","last_fullsized":"2024-03-28T15:10:38.957255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:09.229020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003899156,"stat_sum":{"num_bytes":10797940,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":229,"num_read_kb":5058,"num_write":352,"num_write_kb":10729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"37'10","reported_seq":62,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:34.525786+0000","last_change":"2024-03-28T15:01:31.086634+0000","last_active":"2024-03-28T15:10:34.525786+0000","last_peered":"2024-03-28T15:10:34.525786+0000","last_clean":"2024-03-28T15:10:34.525786+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:10:34.525786+0000","last_undegraded":"2024-03-28T15:10:34.525786+0000","last_fullsized":"2024-03-28T15:10:34.525786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":4136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"37'326","reported_seq":2660,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957715+0000","last_change":"2024-03-28T15:04:56.744447+0000","last_active":"2024-03-28T15:10:38.957715+0000","last_peered":"2024-03-28T15:10:38.957715+0000","last_clean":"2024-03-28T15:10:38.957715+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:10:38.957715+0000","last_undegraded":"2024-03-28T15:10:38.957715+0000","last_fullsized":"2024-03-28T15:10:38.957715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:43.786965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026010120000000002,"stat_sum":{"num_bytes":14545241,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":260,"num_read_kb":2397,"num_write":318,"num_write_kb":18466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":208,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:57.047088+0000","last_change":"2024-03-28T15:01:20.724163+0000","last_active":"2024-03-28T15:09:57.047088+0000","last_peered":"2024-03-28T15:09:57.047088+0000","last_clean":"2024-03-28T15:09:57.047088+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:09:57.047088+0000","last_undegraded":"2024-03-28T15:09:57.047088+0000","last_fullsized":"2024-03-28T15:09:57.047088+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_clean_scrub_stamp":"2024-03-28T15:01:19.548090+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:15.791893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"37'84","reported_seq":137,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:03.245313+0000","last_change":"2024-03-28T15:01:31.085453+0000","last_active":"2024-03-28T15:10:03.245313+0000","last_peered":"2024-03-28T15:10:03.245313+0000","last_clean":"2024-03-28T15:10:03.245313+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:10:03.245313+0000","last_undegraded":"2024-03-28T15:10:03.245313+0000","last_fullsized":"2024-03-28T15:10:03.245313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":85,"num_write_kb":4165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"37'326","reported_seq":2489,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960193+0000","last_change":"2024-03-28T15:04:56.208514+0000","last_active":"2024-03-28T15:10:38.960193+0000","last_peered":"2024-03-28T15:10:38.960193+0000","last_clean":"2024-03-28T15:10:38.960193+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:10:38.960193+0000","last_undegraded":"2024-03-28T15:10:38.960193+0000","last_fullsized":"2024-03-28T15:10:38.960193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:09.229020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002201228,"stat_sum":{"num_bytes":6136325,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":89,"num_read_kb":4109,"num_write":317,"num_write_kb":10222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"37'225","reported_seq":285,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:30.368284+0000","last_change":"2024-03-28T15:01:31.095439+0000","last_active":"2024-03-28T15:10:30.368284+0000","last_peered":"2024-03-28T15:10:30.368284+0000","last_clean":"2024-03-28T15:10:30.368284+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:10:30.368284+0000","last_undegraded":"2024-03-28T15:10:30.368284+0000","last_fullsized":"2024-03-28T15:10:30.368284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":10,"num_read_kb":28674,"num_write":226,"num_write_kb":12578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"37'330","reported_seq":2633,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932587+0000","last_change":"2024-03-28T15:04:56.198312+0000","last_active":"2024-03-28T15:10:38.932587+0000","last_peered":"2024-03-28T15:10:38.932587+0000","last_clean":"2024-03-28T15:10:38.932587+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:10:38.932587+0000","last_undegraded":"2024-03-28T15:10:38.932587+0000","last_fullsized":"2024-03-28T15:10:38.932587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:30.445328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026072840000000001,"stat_sum":{"num_bytes":10172814,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":229,"num_read_kb":5026,"num_write":323,"num_write_kb":10109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'80","reported_seq":209,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:10.281992+0000","last_change":"2024-03-28T15:01:31.093265+0000","last_active":"2024-03-28T15:10:10.281992+0000","last_peered":"2024-03-28T15:10:10.281992+0000","last_clean":"2024-03-28T15:10:10.281992+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:10:10.281992+0000","last_undegraded":"2024-03-28T15:10:10.281992+0000","last_fullsized":"2024-03-28T15:10:10.281992+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534189,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":569,"num_write":78,"num_write_kb":592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"37'335","reported_seq":2426,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937905+0000","last_change":"2024-03-28T15:04:56.293657+0000","last_active":"2024-03-28T15:10:38.937905+0000","last_peered":"2024-03-28T15:10:38.937905+0000","last_clean":"2024-03-28T15:10:38.937905+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:10:38.937905+0000","last_undegraded":"2024-03-28T15:10:38.937905+0000","last_fullsized":"2024-03-28T15:10:38.937905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:56:09.141610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017724730000000001,"stat_sum":{"num_bytes":1382309,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":326,"num_write_kb":1524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"25'82","reported_seq":140,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.264455+0000","last_change":"2024-03-28T15:01:31.094750+0000","last_active":"2024-03-28T15:10:01.264455+0000","last_peered":"2024-03-28T15:10:01.264455+0000","last_clean":"2024-03-28T15:10:01.264455+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:10:01.264455+0000","last_undegraded":"2024-03-28T15:10:01.264455+0000","last_fullsized":"2024-03-28T15:10:01.264455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":80,"num_write_kb":8230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"37'356","reported_seq":2673,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.933205+0000","last_change":"2024-03-28T15:04:56.198147+0000","last_active":"2024-03-28T15:10:38.933205+0000","last_peered":"2024-03-28T15:10:38.933205+0000","last_clean":"2024-03-28T15:10:38.933205+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:10:38.933205+0000","last_undegraded":"2024-03-28T15:10:38.933205+0000","last_fullsized":"2024-03-28T15:10:38.933205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:59.918813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075601100000000001,"stat_sum":{"num_bytes":5557804,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":243,"num_read_kb":4111,"num_write":349,"num_write_kb":5610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"36'123","reported_seq":177,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.244288+0000","last_change":"2024-03-28T15:01:31.093706+0000","last_active":"2024-03-28T15:10:01.244288+0000","last_peered":"2024-03-28T15:10:01.244288+0000","last_clean":"2024-03-28T15:10:01.244288+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:10:01.244288+0000","last_undegraded":"2024-03-28T15:10:01.244288+0000","last_fullsized":"2024-03-28T15:10:01.244288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4096,"num_write":123,"num_write_kb":4157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"37'346","reported_seq":2940,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956753+0000","last_change":"2024-03-28T15:04:56.200588+0000","last_active":"2024-03-28T15:10:38.956753+0000","last_peered":"2024-03-28T15:10:38.956753+0000","last_clean":"2024-03-28T15:10:38.956753+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:10:38.956753+0000","last_undegraded":"2024-03-28T15:10:38.956753+0000","last_fullsized":"2024-03-28T15:10:38.956753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:08.686987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038365610000000001,"stat_sum":{"num_bytes":18167101,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":520,"num_read_kb":17729,"num_write":341,"num_write_kb":26114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"37'1696","reported_seq":1753,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:23.448944+0000","last_change":"2024-03-28T15:01:31.095126+0000","last_active":"2024-03-28T15:10:23.448944+0000","last_peered":"2024-03-28T15:10:23.448944+0000","last_clean":"2024-03-28T15:10:23.448944+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:10:23.448944+0000","last_undegraded":"2024-03-28T15:10:23.448944+0000","last_fullsized":"2024-03-28T15:10:23.448944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1696,"log_dups_size":0,"ondisk_log_size":1696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:21.669535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":12343,"num_write":1697,"num_write_kb":9020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"37'339","reported_seq":2633,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.930142+0000","last_change":"2024-03-28T15:04:56.293414+0000","last_active":"2024-03-28T15:10:38.930142+0000","last_peered":"2024-03-28T15:10:38.930142+0000","last_clean":"2024-03-28T15:10:38.930142+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:10:38.930142+0000","last_undegraded":"2024-03-28T15:10:38.930142+0000","last_fullsized":"2024-03-28T15:10:38.930142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:05:58.503942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017181130000000001,"stat_sum":{"num_bytes":6397362,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":220,"num_read_kb":4361,"num_write":330,"num_write_kb":6420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'146","reported_seq":206,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.479981+0000","last_change":"2024-03-28T15:01:31.093197+0000","last_active":"2024-03-28T15:10:38.479981+0000","last_peered":"2024-03-28T15:10:38.479981+0000","last_clean":"2024-03-28T15:10:38.479981+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:10:38.479981+0000","last_undegraded":"2024-03-28T15:10:38.479981+0000","last_fullsized":"2024-03-28T15:10:38.479981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":239,"num_write":363,"num_write_kb":1821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"37'324","reported_seq":3170,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.946211+0000","last_change":"2024-03-28T15:04:56.208109+0000","last_active":"2024-03-28T15:10:38.946211+0000","last_peered":"2024-03-28T15:10:38.946211+0000","last_clean":"2024-03-28T15:10:38.946211+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:10:38.946211+0000","last_undegraded":"2024-03-28T15:10:38.946211+0000","last_fullsized":"2024-03-28T15:10:38.946211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:30.228263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002114524,"stat_sum":{"num_bytes":15625536,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":772,"num_read_kb":10204,"num_write":317,"num_write_kb":19524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"36'175","reported_seq":235,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:14.520739+0000","last_change":"2024-03-28T15:01:31.088204+0000","last_active":"2024-03-28T15:10:14.520739+0000","last_peered":"2024-03-28T15:10:14.520739+0000","last_clean":"2024-03-28T15:10:14.520739+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:10:14.520739+0000","last_undegraded":"2024-03-28T15:10:14.520739+0000","last_fullsized":"2024-03-28T15:10:14.520739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":4117,"num_write":175,"num_write_kb":4205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"37'353","reported_seq":3392,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932088+0000","last_change":"2024-03-28T15:04:56.198120+0000","last_active":"2024-03-28T15:10:38.932088+0000","last_peered":"2024-03-28T15:10:38.932088+0000","last_clean":"2024-03-28T15:10:38.932088+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:10:38.932088+0000","last_undegraded":"2024-03-28T15:10:38.932088+0000","last_fullsized":"2024-03-28T15:10:38.932088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:21.937841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025149030000000002,"stat_sum":{"num_bytes":28522087,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":966,"num_read_kb":19238,"num_write":341,"num_write_kb":28027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'224","reported_seq":279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.644049+0000","last_change":"2024-03-28T15:01:31.095371+0000","last_active":"2024-03-28T15:10:29.644049+0000","last_peered":"2024-03-28T15:10:29.644049+0000","last_clean":"2024-03-28T15:10:29.644049+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:10:29.644049+0000","last_undegraded":"2024-03-28T15:10:29.644049+0000","last_fullsized":"2024-03-28T15:10:29.644049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":230,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"37'351","reported_seq":2841,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:33.541714+0000","last_change":"2024-03-28T15:04:56.196825+0000","last_active":"2024-03-28T15:10:33.541714+0000","last_peered":"2024-03-28T15:10:33.541714+0000","last_clean":"2024-03-28T15:10:33.541714+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:10:33.541714+0000","last_undegraded":"2024-03-28T15:10:33.541714+0000","last_fullsized":"2024-03-28T15:10:33.541714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:43:14.280586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083411100000000001,"stat_sum":{"num_bytes":27575979,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":416,"num_read_kb":10935,"num_write":349,"num_write_kb":27109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"36'18","reported_seq":71,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.860756+0000","last_change":"2024-03-28T15:01:31.094542+0000","last_active":"2024-03-28T15:10:02.860756+0000","last_peered":"2024-03-28T15:10:02.860756+0000","last_clean":"2024-03-28T15:10:02.860756+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:10:02.860756+0000","last_undegraded":"2024-03-28T15:10:02.860756+0000","last_fullsized":"2024-03-28T15:10:02.860756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":18,"num_write_kb":4204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"37'362","reported_seq":3011,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938795+0000","last_change":"2024-03-28T15:04:56.208858+0000","last_active":"2024-03-28T15:10:38.938795+0000","last_peered":"2024-03-28T15:10:38.938795+0000","last_clean":"2024-03-28T15:10:38.938795+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:10:38.938795+0000","last_undegraded":"2024-03-28T15:10:38.938795+0000","last_fullsized":"2024-03-28T15:10:38.938795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:36.687483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015964149999999999,"stat_sum":{"num_bytes":23553308,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":575,"num_read_kb":17168,"num_write":351,"num_write_kb":27278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"37'29","reported_seq":84,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:14.524755+0000","last_change":"2024-03-28T15:01:31.094440+0000","last_active":"2024-03-28T15:10:14.524755+0000","last_peered":"2024-03-28T15:10:14.524755+0000","last_clean":"2024-03-28T15:10:14.524755+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:10:14.524755+0000","last_undegraded":"2024-03-28T15:10:14.524755+0000","last_fullsized":"2024-03-28T15:10:14.524755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":31,"num_write_kb":4146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"37'329","reported_seq":2882,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937387+0000","last_change":"2024-03-28T15:04:56.208307+0000","last_active":"2024-03-28T15:10:38.937387+0000","last_peered":"2024-03-28T15:10:38.937387+0000","last_clean":"2024-03-28T15:10:38.937387+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:10:38.937387+0000","last_undegraded":"2024-03-28T15:10:38.937387+0000","last_fullsized":"2024-03-28T15:10:38.937387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:08.686987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022492050000000002,"stat_sum":{"num_bytes":15497791,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":479,"num_read_kb":3997,"num_write":321,"num_write_kb":15303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"37'214","reported_seq":279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:39.457500+0000","last_change":"2024-03-28T15:01:31.096352+0000","last_active":"2024-03-28T15:10:39.457500+0000","last_peered":"2024-03-28T15:10:39.457500+0000","last_clean":"2024-03-28T15:10:39.457500+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:10:39.457500+0000","last_undegraded":"2024-03-28T15:10:39.457500+0000","last_fullsized":"2024-03-28T15:10:39.457500+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9811737,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":18,"num_read_kb":28682,"num_write":209,"num_write_kb":9709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"37'352","reported_seq":2837,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.930151+0000","last_change":"2024-03-28T15:04:56.534400+0000","last_active":"2024-03-28T15:10:38.930151+0000","last_peered":"2024-03-28T15:10:38.930151+0000","last_clean":"2024-03-28T15:10:38.930151+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:10:38.930151+0000","last_undegraded":"2024-03-28T15:10:38.930151+0000","last_fullsized":"2024-03-28T15:10:38.930151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:43.521019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001719216,"stat_sum":{"num_bytes":11206715,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":411,"num_read_kb":8309,"num_write":347,"num_write_kb":11133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"37'327","reported_seq":2611,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.955085+0000","last_change":"2024-03-28T15:04:56.200097+0000","last_active":"2024-03-28T15:10:38.955085+0000","last_peered":"2024-03-28T15:10:38.955085+0000","last_clean":"2024-03-28T15:10:38.955085+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:10:38.955085+0000","last_undegraded":"2024-03-28T15:10:38.955085+0000","last_fullsized":"2024-03-28T15:10:38.955085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:17:15.635128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036538370000000001,"stat_sum":{"num_bytes":7862984,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":211,"num_read_kb":4119,"num_write":311,"num_write_kb":11936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"26'24","reported_seq":85,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.901412+0000","last_change":"2024-03-28T15:01:31.094281+0000","last_active":"2024-03-28T15:10:01.901412+0000","last_peered":"2024-03-28T15:10:01.901412+0000","last_clean":"2024-03-28T15:10:01.901412+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:10:01.901412+0000","last_undegraded":"2024-03-28T15:10:01.901412+0000","last_fullsized":"2024-03-28T15:10:01.901412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":24,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"37'344","reported_seq":3680,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956194+0000","last_change":"2024-03-28T15:04:56.200342+0000","last_active":"2024-03-28T15:10:38.956194+0000","last_peered":"2024-03-28T15:10:38.956194+0000","last_clean":"2024-03-28T15:10:38.956194+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:10:38.956194+0000","last_undegraded":"2024-03-28T15:10:38.956194+0000","last_fullsized":"2024-03-28T15:10:38.956194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:51:56.779182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037913880000000001,"stat_sum":{"num_bytes":22932535,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":1263,"num_read_kb":23613,"num_write":335,"num_write_kb":26620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"37'170","reported_seq":237,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.643756+0000","last_change":"2024-03-28T15:01:31.093773+0000","last_active":"2024-03-28T15:10:29.643756+0000","last_peered":"2024-03-28T15:10:29.643756+0000","last_clean":"2024-03-28T15:10:29.643756+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:10:29.643756+0000","last_undegraded":"2024-03-28T15:10:29.643756+0000","last_fullsized":"2024-03-28T15:10:29.643756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:54.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":36885,"num_write":163,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"37'319","reported_seq":2904,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.943319+0000","last_change":"2024-03-28T15:04:56.197860+0000","last_active":"2024-03-28T15:10:38.943319+0000","last_peered":"2024-03-28T15:10:38.943319+0000","last_clean":"2024-03-28T15:10:38.943319+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:10:38.943319+0000","last_undegraded":"2024-03-28T15:10:38.943319+0000","last_fullsized":"2024-03-28T15:10:38.943319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:22.085679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023175320000000002,"stat_sum":{"num_bytes":14729150,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":511,"num_read_kb":13471,"num_write":311,"num_write_kb":14544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"37'1099","reported_seq":1154,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.257121+0000","last_change":"2024-03-28T15:01:31.096725+0000","last_active":"2024-03-28T15:10:01.257121+0000","last_peered":"2024-03-28T15:10:01.257121+0000","last_clean":"2024-03-28T15:10:01.257121+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:10:01.257121+0000","last_undegraded":"2024-03-28T15:10:01.257121+0000","last_fullsized":"2024-03-28T15:10:01.257121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":1093,"num_write_kb":5019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"37'346","reported_seq":2920,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935556+0000","last_change":"2024-03-28T15:04:56.207406+0000","last_active":"2024-03-28T15:10:38.935556+0000","last_peered":"2024-03-28T15:10:38.935556+0000","last_clean":"2024-03-28T15:10:38.935556+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:10:38.935556+0000","last_undegraded":"2024-03-28T15:10:38.935556+0000","last_fullsized":"2024-03-28T15:10:38.935556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:57:18.708712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015914119999999999,"stat_sum":{"num_bytes":14776390,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":500,"num_read_kb":7280,"num_write":341,"num_write_kb":14609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"36'104","reported_seq":172,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:03.114633+0000","last_change":"2024-03-28T15:01:31.095351+0000","last_active":"2024-03-28T15:10:03.114633+0000","last_peered":"2024-03-28T15:10:03.114633+0000","last_clean":"2024-03-28T15:10:03.114633+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:10:03.114633+0000","last_undegraded":"2024-03-28T15:10:03.114633+0000","last_fullsized":"2024-03-28T15:10:03.114633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:21:07.910058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":20500,"num_write":104,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":940897971,"num_objects":10589,"num_object_clones":0,"num_object_copies":21178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10589,"num_whiteouts":0,"num_read":30680,"num_read_kb":666406,"num_write":21173,"num_write_kb":1063512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1935458304,"data_stored":1880255846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21704,"ondisk_log_size":21704,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":182313860,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":492,"num_read_kb":424818,"num_write":6917,"num_write_kb":184630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364691456,"data_stored":364627720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6716,"ondisk_log_size":6716,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574965,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425508,"kb_used_data":336092,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":93946332,"statfs":{"total":96636764160,"available":96201043968,"internally_reserved":0,"allocated":344158208,"data_stored":335751183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":91552202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574966,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294192,"kb_used_data":240808,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94077648,"statfs":{"total":96636764160,"available":96335511552,"internally_reserved":0,"allocated":246587392,"data_stored":241270326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574966,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354032,"kb_used_data":308968,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94017808,"statfs":{"total":96636764160,"available":96274235392,"internally_reserved":0,"allocated":316383232,"data_stored":308090025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574966,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":387780,"kb_used_data":338556,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93984060,"statfs":{"total":96636764160,"available":96239677440,"internally_reserved":0,"allocated":346681344,"data_stored":338235386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574965,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405916,"kb_used_data":360852,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93965924,"statfs":{"total":96636764160,"available":96221106176,"internally_reserved":0,"allocated":369512448,"data_stored":360982077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181040,"kb_used_data":149864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94190800,"statfs":{"total":96636764160,"available":96451379200,"internally_reserved":0,"allocated":153460736,"data_stored":148434971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267092,"kb_used_data":217868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94104748,"statfs":{"total":96636764160,"available":96363261952,"internally_reserved":0,"allocated":223096832,"data_stored":217893056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574965,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368928,"kb_used_data":297560,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94002912,"statfs":{"total":96636764160,"available":96258981888,"internally_reserved":0,"allocated":304701440,"data_stored":296790976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50331808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21508096,"data_stored":21505709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":34099200,"data_stored":34089293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51769344,"data_stored":51754903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50332320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":72732672,"data_stored":72726879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261738496,"data_stored":254075801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172343296,"data_stored":167380119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131538944,"data_stored":126748133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334999552,"data_stored":326711655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":294531072,"data_stored":286332021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":274022400,"data_stored":265965766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":195833856,"data_stored":190756877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":270450688,"data_stored":262285474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:10:46.040 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:10:46.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.176+0000 7f4281d93640 1 Processor -- start 2024-03-28T15:10:46.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.176+0000 7f4281d93640 1 -- start start 2024-03-28T15:10:46.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f4281d93640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f427c06be00 0x7f427c0c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:46.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f4281d93640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f427c0c95e0 0x7f427c15aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:46.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f4281d93640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f427c06c870 con 0x7f427c0c9b10 2024-03-28T15:10:46.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f4281d93640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f427c06b540 con 0x7f427c0c95e0 2024-03-28T15:10:46.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f4281d93640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f427c06cb50 con 0x7f427c06be00 2024-03-28T15:10:46.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f427affd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f427c0c95e0 0x7f427c15aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:46.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f427b7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f427c06be00 0x7f427c0c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:46.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f427affd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f427c0c95e0 0x7f427c15aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36680/0 (socket says 172.21.15.186:36680) 2024-03-28T15:10:46.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f427affd640 1 -- 172.21.15.186:0/3851483902 learned_addr learned my addr 172.21.15.186:0/3851483902 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:46.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f427a7fc640 1 -- 172.21.15.186:0/3851483902 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3372700682 0 0) 0x7f427c06c870 con 0x7f427c0c9b10 2024-03-28T15:10:46.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f427b7fe640 1 -- 172.21.15.186:0/3851483902 >> v2:172.21.15.186:3300/0 conn(0x7f427c0c95e0 msgr2=0x7f427c15aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:46.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f427b7fe640 1 --2- 172.21.15.186:0/3851483902 >> v2:172.21.15.186:3300/0 conn(0x7f427c0c95e0 0x7f427c15aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f427b7fe640 1 -- 172.21.15.186:0/3851483902 >> v1:172.21.15.191:6789/0 conn(0x7f427c0c9b10 legacy=0x7f427c067280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:46.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.177+0000 7f427b7fe640 1 -- 172.21.15.186:0/3851483902 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427c15df80 con 0x7f427c06be00 2024-03-28T15:10:46.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.178+0000 7f427b7fe640 1 --2- 172.21.15.186:0/3851483902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f427c06be00 0x7f427c0c7a80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f426c0097a0 tx=0x7f426c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f5bda7431583c00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:46.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.178+0000 7f427a7fc640 1 -- 172.21.15.186:0/3851483902 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c034070 con 0x7f427c06be00 2024-03-28T15:10:46.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.178+0000 7f427a7fc640 1 -- 172.21.15.186:0/3851483902 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426c030610 con 0x7f427c06be00 2024-03-28T15:10:46.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.178+0000 7f427a7fc640 1 -- 172.21.15.186:0/3851483902 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c0309e0 con 0x7f427c06be00 2024-03-28T15:10:46.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.179+0000 7f4281d93640 1 -- 172.21.15.186:0/3851483902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f427c06be00 msgr2=0x7f427c0c7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:46.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.179+0000 7f4281d93640 1 --2- 172.21.15.186:0/3851483902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f427c06be00 0x7f427c0c7a80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f426c0097a0 tx=0x7f426c02b960 comp rx=0 tx=0).stop 2024-03-28T15:10:46.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.179+0000 7f4281d93640 1 -- 172.21.15.186:0/3851483902 shutdown_connections 2024-03-28T15:10:46.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.179+0000 7f4281d93640 1 --2- 172.21.15.186:0/3851483902 >> v2:172.21.15.186:3300/0 conn(0x7f427c0c95e0 0x7f427c15aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.179+0000 7f4281d93640 1 --2- 172.21.15.186:0/3851483902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f427c06be00 0x7f427c0c7a80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.179+0000 7f4281d93640 1 -- 172.21.15.186:0/3851483902 >> 172.21.15.186:0/3851483902 conn(0x7f427c06e970 msgr2=0x7f427c06ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:46.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.179+0000 7f4281d93640 1 -- 172.21.15.186:0/3851483902 shutdown_connections 2024-03-28T15:10:46.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.179+0000 7f4281d93640 1 -- 172.21.15.186:0/3851483902 wait complete. 2024-03-28T15:10:46.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.180+0000 7f4281d93640 1 Processor -- start 2024-03-28T15:10:46.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.180+0000 7f4281d93640 1 -- start start 2024-03-28T15:10:46.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f4281d93640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c06be00 0x7f427c159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:46.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f4281d93640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f427c0c95e0 0x7f427c0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:46.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f4281d93640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f427c0c9b10 0x7f427c153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:46.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f4281d93640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f427c15e820 con 0x7f427c06be00 2024-03-28T15:10:46.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f4281d93640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f427c15e540 con 0x7f427c0c9b10 2024-03-28T15:10:46.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f4281d93640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f427c06c530 con 0x7f427c0c95e0 2024-03-28T15:10:46.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f427b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c06be00 0x7f427c159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:46.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f427b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c06be00 0x7f427c159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36684/0 (socket says 172.21.15.186:36684) 2024-03-28T15:10:46.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f427b7fe640 1 -- 172.21.15.186:0/3077081672 learned_addr learned my addr 172.21.15.186:0/3077081672 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:46.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f427affd640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f427c0c95e0 0x7f427c0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:46.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f427bfff640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f427c0c9b10 0x7f427c153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:46.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f427b7fe640 1 -- 172.21.15.186:0/3077081672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f427c0c95e0 msgr2=0x7f427c0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:46.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f427b7fe640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f427c0c95e0 0x7f427c0cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f427b7fe640 1 -- 172.21.15.186:0/3077081672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f427c0c9b10 msgr2=0x7f427c153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:46.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f427b7fe640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f427c0c9b10 0x7f427c153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f427b7fe640 1 -- 172.21.15.186:0/3077081672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427c1f00e0 con 0x7f427c06be00 2024-03-28T15:10:46.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f427b7fe640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c06be00 0x7f427c159b40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f426c02bf70 tx=0x7f426c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:46.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f4263fff640 1 -- 172.21.15.186:0/3077081672 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c034040 con 0x7f427c06be00 2024-03-28T15:10:46.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.181+0000 7f4263fff640 1 -- 172.21.15.186:0/3077081672 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426c030410 con 0x7f427c06be00 2024-03-28T15:10:46.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.182+0000 7f4281d93640 1 -- 172.21.15.186:0/3077081672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427c1f03c0 con 0x7f427c06be00 2024-03-28T15:10:46.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.182+0000 7f4263fff640 1 -- 172.21.15.186:0/3077081672 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c039c30 con 0x7f427c06be00 2024-03-28T15:10:46.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.182+0000 7f4281d93640 1 -- 172.21.15.186:0/3077081672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427c1f0980 con 0x7f427c06be00 2024-03-28T15:10:46.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.182+0000 7f4281d93640 1 -- 172.21.15.186:0/3077081672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f427c0708b0 con 0x7f427c06be00 2024-03-28T15:10:46.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.183+0000 7f4263fff640 1 -- 172.21.15.186:0/3077081672 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f426c037070 con 0x7f427c06be00 2024-03-28T15:10:46.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.183+0000 7f4263fff640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f426806c530 0x7f426806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:46.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.183+0000 7f427affd640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f426806c530 0x7f426806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:46.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.183+0000 7f4263fff640 1 -- 172.21.15.186:0/3077081672 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f426c0ad7c0 con 0x7f427c06be00 2024-03-28T15:10:46.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.184+0000 7f427affd640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f426806c530 0x7f426806e9f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f427c0b1f80 tx=0x7f4270009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:46.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.188+0000 7f4263fff640 1 -- 172.21.15.186:0/3077081672 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f427c0708b0 con 0x7f427c06be00 2024-03-28T15:10:46.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.364+0000 7f4281d93640 1 -- 172.21.15.186:0/3077081672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f427c069500 con 0x7f427c06be00 2024-03-28T15:10:46.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.365+0000 7f4263fff640 1 -- 172.21.15.186:0/3077081672 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+17572 (secure 0 0 0) 0x7f426c07b4f0 con 0x7f427c06be00 2024-03-28T15:10:46.366 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:46.366 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":37,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:09:56.237674+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.191:6834/1992706523":"2024-03-29T15:08:31.220829+0000","172.21.15.191:6837/3037586083":"2024-03-29T15:09:56.237602+0000","172.21.15.191:6835/3037586083":"2024-03-29T15:09:56.237602+0000","172.21.15.191:6837/3828930138":"2024-03-29T15:06:06.202259+0000","172.21.15.191:6835/3828930138":"2024-03-29T15:06:06.202259+0000","172.21.15.191:6832/1992706523":"2024-03-29T15:08:31.220829+0000","172.21.15.191:6834/185536353":"2024-03-29T15:03:46.182756+0000","172.21.15.191:6832/185536353":"2024-03-29T15:03:46.182756+0000","172.21.15.191:6837/2813406930":"2024-03-29T15:02:11.169402+0000","172.21.15.191:6835/2813406930":"2024-03-29T15:02:11.169402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:10:46.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 -- 172.21.15.186:0/3077081672 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f426806c530 msgr2=0x7f426806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:46.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f426806c530 0x7f426806e9f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f427c0b1f80 tx=0x7f4270009240 comp rx=0 tx=0).stop 2024-03-28T15:10:46.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 -- 172.21.15.186:0/3077081672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c06be00 msgr2=0x7f427c159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:46.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c06be00 0x7f427c159b40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f426c02bf70 tx=0x7f426c0046c0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 -- 172.21.15.186:0/3077081672 shutdown_connections 2024-03-28T15:10:46.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f426806c530 0x7f426806e9f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f427c0c9b10 0x7f427c153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.368 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f427c0c95e0 0x7f427c0cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 --2- 172.21.15.186:0/3077081672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c06be00 0x7f427c159b40 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:46.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 -- 172.21.15.186:0/3077081672 >> 172.21.15.186:0/3077081672 conn(0x7f427c06e970 msgr2=0x7f427c0c2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:46.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 -- 172.21.15.186:0/3077081672 shutdown_connections 2024-03-28T15:10:46.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:46.367+0000 7f4281d93640 1 -- 172.21.15.186:0/3077081672 wait complete. 2024-03-28T15:10:47.379 INFO:tasks.ceph:Scrubbing osd.0 2024-03-28T15:10:47.380 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-28T15:10:47.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.492+0000 7f19a9a83640 1 Processor -- start 2024-03-28T15:10:47.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.492+0000 7f19a9a83640 1 -- start start 2024-03-28T15:10:47.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.492+0000 7f19a9a83640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f19a415af20 0x7f19a40b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f19a9a83640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f19a4073d40 0x7f19a4161570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f19a9a83640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19a406dda0 con 0x7f19a40b1750 2024-03-28T15:10:47.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f19a9a83640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f19a406d2b0 con 0x7f19a4073d40 2024-03-28T15:10:47.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f19a9a83640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f19a415a380 con 0x7f19a415af20 2024-03-28T15:10:47.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f19a3fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f19a4073d40 0x7f19a4161570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f19a3fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f19a4073d40 0x7f19a4161570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36686/0 (socket says 172.21.15.186:36686) 2024-03-28T15:10:47.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f19a3fff640 1 -- 172.21.15.186:0/1358186795 learned_addr learned my addr 172.21.15.186:0/1358186795 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:47.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f199bfff640 1 --2- 172.21.15.186:0/1358186795 >> v2:172.21.15.191:3300/0 conn(0x7f19a415af20 0x7f19a40b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f19a3fff640 1 -- 172.21.15.186:0/1358186795 >> v2:172.21.15.191:3300/0 conn(0x7f19a415af20 msgr2=0x7f19a40b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:47.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f19a3fff640 1 --2- 172.21.15.186:0/1358186795 >> v2:172.21.15.191:3300/0 conn(0x7f19a415af20 0x7f19a40b1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f19a3fff640 1 -- 172.21.15.186:0/1358186795 >> v1:172.21.15.191:6789/0 conn(0x7f19a40b1750 legacy=0x7f19a415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.493+0000 7f19a3fff640 1 -- 172.21.15.186:0/1358186795 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a406d500 con 0x7f19a4073d40 2024-03-28T15:10:47.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.494+0000 7f19a3fff640 1 --2- 172.21.15.186:0/1358186795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4073d40 0x7f19a4161570 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f19a406d430 tx=0x7f199c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a980700638762eca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:47.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.494+0000 7f19a2ffd640 1 -- 172.21.15.186:0/1358186795 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199c02cb00 con 0x7f19a4073d40 2024-03-28T15:10:47.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.494+0000 7f19a2ffd640 1 -- 172.21.15.186:0/1358186795 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f199c02cca0 con 0x7f19a4073d40 2024-03-28T15:10:47.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.494+0000 7f19a2ffd640 1 -- 172.21.15.186:0/1358186795 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199c034610 con 0x7f19a4073d40 2024-03-28T15:10:47.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.494+0000 7f19a9a83640 1 -- 172.21.15.186:0/1358186795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4073d40 msgr2=0x7f19a4161570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.494+0000 7f19a9a83640 1 --2- 172.21.15.186:0/1358186795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4073d40 0x7f19a4161570 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f19a406d430 tx=0x7f199c02bf60 comp rx=0 tx=0).stop 2024-03-28T15:10:47.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.495+0000 7f19a9a83640 1 -- 172.21.15.186:0/1358186795 shutdown_connections 2024-03-28T15:10:47.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.495+0000 7f19a9a83640 1 --2- 172.21.15.186:0/1358186795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4073d40 0x7f19a4161570 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.495+0000 7f19a9a83640 1 --2- 172.21.15.186:0/1358186795 >> v2:172.21.15.191:3300/0 conn(0x7f19a415af20 0x7f19a40b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.495+0000 7f19a9a83640 1 -- 172.21.15.186:0/1358186795 >> 172.21.15.186:0/1358186795 conn(0x7f19a40c8a20 msgr2=0x7f19a40c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:47.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.495+0000 7f19a9a83640 1 -- 172.21.15.186:0/1358186795 shutdown_connections 2024-03-28T15:10:47.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.495+0000 7f19a9a83640 1 -- 172.21.15.186:0/1358186795 wait complete. 2024-03-28T15:10:47.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.496+0000 7f19a9a83640 1 Processor -- start 2024-03-28T15:10:47.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.496+0000 7f19a9a83640 1 -- start start 2024-03-28T15:10:47.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.496+0000 7f19a9a83640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4073d40 0x7f19a4158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.496+0000 7f19a9a83640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19a40b1750 0x7f19a4159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.496+0000 7f19a9a83640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19a415af20 0x7f19a414ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.496+0000 7f19a9a83640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f19a4151320 con 0x7f19a4073d40 2024-03-28T15:10:47.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.496+0000 7f19a9a83640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f19a406dac0 con 0x7f19a40b1750 2024-03-28T15:10:47.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.496+0000 7f19a9a83640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f19a4073500 con 0x7f19a415af20 2024-03-28T15:10:47.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.496+0000 7f19a37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4073d40 0x7f19a4158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4073d40 0x7f19a4158ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36692/0 (socket says 172.21.15.186:36692) 2024-03-28T15:10:47.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a37fe640 1 -- 172.21.15.186:0/2847142196 learned_addr learned my addr 172.21.15.186:0/2847142196 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:47.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f199bfff640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19a40b1750 0x7f19a4159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a3fff640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19a415af20 0x7f19a414ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a37fe640 1 -- 172.21.15.186:0/2847142196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19a415af20 msgr2=0x7f19a414ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a37fe640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19a415af20 0x7f19a414ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a37fe640 1 -- 172.21.15.186:0/2847142196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19a40b1750 msgr2=0x7f19a4159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a37fe640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19a40b1750 0x7f19a4159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a37fe640 1 -- 172.21.15.186:0/2847142196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a40cabc0 con 0x7f19a4073d40 2024-03-28T15:10:47.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a37fe640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4073d40 0x7f19a4158ba0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f199400c400 tx=0x7f199400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:47.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a0ff9640 1 -- 172.21.15.186:0/2847142196 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19940052d0 con 0x7f19a4073d40 2024-03-28T15:10:47.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a0ff9640 1 -- 172.21.15.186:0/2847142196 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1994005490 con 0x7f19a4073d40 2024-03-28T15:10:47.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a0ff9640 1 -- 172.21.15.186:0/2847142196 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1994007260 con 0x7f19a4073d40 2024-03-28T15:10:47.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a9a83640 1 -- 172.21.15.186:0/2847142196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19a40caea0 con 0x7f19a4073d40 2024-03-28T15:10:47.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.497+0000 7f19a9a83640 1 -- 172.21.15.186:0/2847142196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19a40cb460 con 0x7f19a4073d40 2024-03-28T15:10:47.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.498+0000 7f19a9a83640 1 -- 172.21.15.186:0/2847142196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f19a414b600 con 0x7f19a4073d40 2024-03-28T15:10:47.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.498+0000 7f19a0ff9640 1 -- 172.21.15.186:0/2847142196 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1994007400 con 0x7f19a4073d40 2024-03-28T15:10:47.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.499+0000 7f19a0ff9640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f198406c530 0x7f198406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.499+0000 7f19a0ff9640 1 -- 172.21.15.186:0/2847142196 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f199408adc0 con 0x7f19a4073d40 2024-03-28T15:10:47.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.499+0000 7f199bfff640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f198406c530 0x7f198406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.499+0000 7f19a0ff9640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] conn(0x7f1984074bd0 0x7f1984077050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.499+0000 7f19a0ff9640 1 -- 172.21.15.186:0/2847142196 --> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1994009e50 con 0x7f1984074bd0 2024-03-28T15:10:47.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.499+0000 7f19a0ff9640 1 -- 172.21.15.186:0/2847142196 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f199408d4c0 con 0x7f19a4073d40 2024-03-28T15:10:47.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.499+0000 7f19a3fff640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] conn(0x7f1984074bd0 0x7f1984077050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.499+0000 7f199bfff640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f198406c530 0x7f198406e9f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f19a41e6ab0 tx=0x7f198c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:47.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.500+0000 7f19a3fff640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] conn(0x7f1984074bd0 0x7f1984077050 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:47.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.500+0000 7f19a0ff9640 1 -- 172.21.15.186:0/2847142196 <== osd.0 v2:172.21.15.186:6824/2448652000 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1994009e50 con 0x7f1984074bd0 2024-03-28T15:10:47.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.513+0000 7f19a9a83640 1 -- 172.21.15.186:0/2847142196 --> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f19a4069570 con 0x7f1984074bd0 2024-03-28T15:10:47.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.513+0000 7f19a0ff9640 1 -- 172.21.15.186:0/2847142196 <== osd.0 v2:172.21.15.186:6824/2448652000 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f19a4069570 con 0x7f1984074bd0 2024-03-28T15:10:47.514 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-28T15:10:47.514 INFO:teuthology.orchestra.run.smithi186.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T15:10:47.514 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-28T15:10:47.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.514+0000 7f19a9a83640 1 -- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] conn(0x7f1984074bd0 msgr2=0x7f1984077050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.514+0000 7f19a9a83640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:6824/2448652000,v1:172.21.15.186:6825/2448652000] conn(0x7f1984074bd0 0x7f1984077050 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.514+0000 7f19a9a83640 1 -- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f198406c530 msgr2=0x7f198406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.514+0000 7f19a9a83640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f198406c530 0x7f198406e9f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f19a41e6ab0 tx=0x7f198c009190 comp rx=0 tx=0).stop 2024-03-28T15:10:47.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.514+0000 7f19a9a83640 1 -- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4073d40 msgr2=0x7f19a4158ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.514+0000 7f19a9a83640 1 --2- 172.21.15.186:0/2847142196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4073d40 0x7f19a4158ba0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f199400c400 tx=0x7f199400c8c0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.514+0000 7f19a3fff640 1 -- 172.21.15.186:0/2847142196 reap_dead start 2024-03-28T15:10:47.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.514+0000 7f19a9a83640 1 -- 172.21.15.186:0/2847142196 shutdown_connections 2024-03-28T15:10:47.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.514+0000 7f19a9a83640 1 -- 172.21.15.186:0/2847142196 >> 172.21.15.186:0/2847142196 conn(0x7f19a40c8a20 msgr2=0x7f19a40c1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:47.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.514+0000 7f19a9a83640 1 -- 172.21.15.186:0/2847142196 shutdown_connections 2024-03-28T15:10:47.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.514+0000 7f19a9a83640 1 -- 172.21.15.186:0/2847142196 wait complete. 2024-03-28T15:10:47.525 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-28T15:10:47.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.627+0000 7fcd22084640 1 Processor -- start 2024-03-28T15:10:47.628 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.627+0000 7fcd22084640 1 -- start start 2024-03-28T15:10:47.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.628+0000 7fcd22084640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcd1c15a7c0 0x7fcd1c0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.628+0000 7fcd22084640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd1c0cb6e0 con 0x7fcd1c154b60 2024-03-28T15:10:47.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.628+0000 7fcd22084640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd1c15b760 con 0x7fcd1c071a80 2024-03-28T15:10:47.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.628+0000 7fcd22084640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7fcd1c072820 con 0x7fcd1c15a7c0 2024-03-28T15:10:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.628+0000 7fcd1b7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fcd1c154b60 0x7fcd1c0cafd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:52754/0 (socket says 172.21.15.186:52754) 2024-03-28T15:10:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.628+0000 7fcd1b7fe640 1 -- 172.21.15.186:0/3785323 learned_addr learned my addr 172.21.15.186:0/3785323 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.628+0000 7fcd1affd640 1 --2- 172.21.15.186:0/3785323 >> v2:172.21.15.191:3300/0 conn(0x7fcd1c15a7c0 0x7fcd1c0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.628+0000 7fcd1affd640 1 -- 172.21.15.186:0/3785323 >> v1:172.21.15.191:6789/0 conn(0x7fcd1c071a80 legacy=0x7fcd1c165140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.628+0000 7fcd1affd640 1 -- 172.21.15.186:0/3785323 >> v1:172.21.15.186:6789/0 conn(0x7fcd1c154b60 legacy=0x7fcd1c0cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.628+0000 7fcd1a7fc640 1 -- 172.21.15.186:0/3785323 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2435153221 0 0) 0x7fcd1c0cb6e0 con 0x7fcd1c154b60 2024-03-28T15:10:47.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.628+0000 7fcd1affd640 1 -- 172.21.15.186:0/3785323 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1c15aec0 con 0x7fcd1c15a7c0 2024-03-28T15:10:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.629+0000 7fcd1affd640 1 --2- 172.21.15.186:0/3785323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd1c15a7c0 0x7fcd1c0ccf10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcd100097a0 tx=0x7fcd1002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0c91b2483084894 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.629+0000 7fcd1a7fc640 1 -- 172.21.15.186:0/3785323 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1002f810 con 0x7fcd1c15a7c0 2024-03-28T15:10:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.629+0000 7fcd1a7fc640 1 -- 172.21.15.186:0/3785323 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd1002f9b0 con 0x7fcd1c15a7c0 2024-03-28T15:10:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.629+0000 7fcd1a7fc640 1 -- 172.21.15.186:0/3785323 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1002fd80 con 0x7fcd1c15a7c0 2024-03-28T15:10:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.630+0000 7fcd22084640 1 -- 172.21.15.186:0/3785323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd1c15a7c0 msgr2=0x7fcd1c0ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.630+0000 7fcd22084640 1 --2- 172.21.15.186:0/3785323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd1c15a7c0 0x7fcd1c0ccf10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcd100097a0 tx=0x7fcd1002b9f0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.630+0000 7fcd22084640 1 -- 172.21.15.186:0/3785323 shutdown_connections 2024-03-28T15:10:47.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.630+0000 7fcd22084640 1 --2- 172.21.15.186:0/3785323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd1c15a7c0 0x7fcd1c0ccf10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.630+0000 7fcd22084640 1 -- 172.21.15.186:0/3785323 >> 172.21.15.186:0/3785323 conn(0x7fcd1c073b80 msgr2=0x7fcd1c06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:47.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.630+0000 7fcd22084640 1 -- 172.21.15.186:0/3785323 shutdown_connections 2024-03-28T15:10:47.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.630+0000 7fcd22084640 1 -- 172.21.15.186:0/3785323 wait complete. 2024-03-28T15:10:47.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.631+0000 7fcd22084640 1 Processor -- start 2024-03-28T15:10:47.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.631+0000 7fcd22084640 1 -- start start 2024-03-28T15:10:47.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd22084640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd1c071a80 0x7fcd1c144db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd22084640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c154b60 0x7fcd1c0c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd22084640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd1c15a7c0 0x7fcd1c15df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd22084640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c0cc710 con 0x7fcd1c154b60 2024-03-28T15:10:47.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd22084640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c15b480 con 0x7fcd1c15a7c0 2024-03-28T15:10:47.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd22084640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fcd1c15abe0 con 0x7fcd1c071a80 2024-03-28T15:10:47.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd1affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c154b60 0x7fcd1c0c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd1affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c154b60 0x7fcd1c0c5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36694/0 (socket says 172.21.15.186:36694) 2024-03-28T15:10:47.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd1affd640 1 -- 172.21.15.186:0/916976934 learned_addr learned my addr 172.21.15.186:0/916976934 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:47.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd1b7fe640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd1c071a80 0x7fcd1c144db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd1bfff640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd1c15a7c0 0x7fcd1c15df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd1affd640 1 -- 172.21.15.186:0/916976934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd1c071a80 msgr2=0x7fcd1c144db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd1affd640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd1c071a80 0x7fcd1c144db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd1affd640 1 -- 172.21.15.186:0/916976934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd1c15a7c0 msgr2=0x7fcd1c15df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd1affd640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd1c15a7c0 0x7fcd1c15df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.632+0000 7fcd1affd640 1 -- 172.21.15.186:0/916976934 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1c0c53c0 con 0x7fcd1c154b60 2024-03-28T15:10:47.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.633+0000 7fcd1affd640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c154b60 0x7fcd1c0c5b50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fcd10000c00 tx=0x7fcd100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:47.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.633+0000 7fcd03fff640 1 -- 172.21.15.186:0/916976934 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd10046070 con 0x7fcd1c154b60 2024-03-28T15:10:47.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.633+0000 7fcd03fff640 1 -- 172.21.15.186:0/916976934 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd10037c40 con 0x7fcd1c154b60 2024-03-28T15:10:47.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.633+0000 7fcd22084640 1 -- 172.21.15.186:0/916976934 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd1c1452f0 con 0x7fcd1c154b60 2024-03-28T15:10:47.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.633+0000 7fcd03fff640 1 -- 172.21.15.186:0/916976934 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1002fe60 con 0x7fcd1c154b60 2024-03-28T15:10:47.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.633+0000 7fcd22084640 1 -- 172.21.15.186:0/916976934 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd1c0c5630 con 0x7fcd1c154b60 2024-03-28T15:10:47.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.634+0000 7fcd22084640 1 -- 172.21.15.186:0/916976934 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd1c0686d0 con 0x7fcd1c154b60 2024-03-28T15:10:47.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.634+0000 7fcd03fff640 1 -- 172.21.15.186:0/916976934 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fcd10049070 con 0x7fcd1c154b60 2024-03-28T15:10:47.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.635+0000 7fcd03fff640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0806c560 0x7fcd0806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.635+0000 7fcd03fff640 1 -- 172.21.15.186:0/916976934 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fcd100acfc0 con 0x7fcd1c154b60 2024-03-28T15:10:47.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.635+0000 7fcd1b7fe640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0806c560 0x7fcd0806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.635+0000 7fcd1b7fe640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0806c560 0x7fcd0806ea20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c007f70 tx=0x7fcd0c007f00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:47.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.639+0000 7fcd03fff640 1 -- 172.21.15.186:0/916976934 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fcd1c1452f0 con 0x7fcd1c154b60 2024-03-28T15:10:47.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.816+0000 7fcd22084640 1 -- 172.21.15.186:0/916976934 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fcd1c0cb910 con 0x7fcd0806c560 2024-03-28T15:10:47.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.816+0000 7fcd03fff640 1 -- 172.21.15.186:0/916976934 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcd1c0cb910 con 0x7fcd0806c560 2024-03-28T15:10:47.817 INFO:teuthology.orchestra.run.smithi186.stderr:instructed osd(s) 0 to deep-scrub 2024-03-28T15:10:47.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.818+0000 7fcd22084640 1 -- 172.21.15.186:0/916976934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0806c560 msgr2=0x7fcd0806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.818+0000 7fcd22084640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0806c560 0x7fcd0806ea20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c007f70 tx=0x7fcd0c007f00 comp rx=0 tx=0).stop 2024-03-28T15:10:47.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.818+0000 7fcd22084640 1 -- 172.21.15.186:0/916976934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c154b60 msgr2=0x7fcd1c0c5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.818+0000 7fcd22084640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c154b60 0x7fcd1c0c5b50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fcd10000c00 tx=0x7fcd100046c0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.818+0000 7fcd22084640 1 -- 172.21.15.186:0/916976934 shutdown_connections 2024-03-28T15:10:47.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.818+0000 7fcd22084640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcd0806c560 0x7fcd0806ea20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.818+0000 7fcd22084640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd1c15a7c0 0x7fcd1c15df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.818+0000 7fcd22084640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c154b60 0x7fcd1c0c5b50 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.818+0000 7fcd22084640 1 --2- 172.21.15.186:0/916976934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd1c071a80 0x7fcd1c144db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.818+0000 7fcd22084640 1 -- 172.21.15.186:0/916976934 >> 172.21.15.186:0/916976934 conn(0x7fcd1c073b80 msgr2=0x7fcd1c1584a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:47.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.819+0000 7fcd22084640 1 -- 172.21.15.186:0/916976934 shutdown_connections 2024-03-28T15:10:47.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.819+0000 7fcd22084640 1 -- 172.21.15.186:0/916976934 wait complete. 2024-03-28T15:10:47.831 INFO:tasks.ceph:Scrubbing osd.1 2024-03-28T15:10:47.831 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-28T15:10:47.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.935+0000 7f418e08a640 1 Processor -- start 2024-03-28T15:10:47.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.935+0000 7f418e08a640 1 -- start start 2024-03-28T15:10:47.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.936+0000 7f418e08a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f418815e850 0x7f41880cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.936+0000 7f418e08a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41880cc180 0x7f41880c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.936+0000 7f418e08a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41880cac20 con 0x7f4188154b10 2024-03-28T15:10:47.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.936+0000 7f418e08a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f41880cc660 con 0x7f41880cc180 2024-03-28T15:10:47.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.936+0000 7f418e08a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f41880cbf00 con 0x7f418815e850 2024-03-28T15:10:47.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.936+0000 7f4187fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41880cc180 0x7f41880c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.936+0000 7f4187fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41880cc180 0x7f41880c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36704/0 (socket says 172.21.15.186:36704) 2024-03-28T15:10:47.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.936+0000 7f4187fff640 1 -- 172.21.15.186:0/1249731925 learned_addr learned my addr 172.21.15.186:0/1249731925 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:47.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.936+0000 7f41877fe640 1 --1- 172.21.15.186:0/1249731925 >> v1:172.21.15.191:6789/0 conn(0x7f4188154b10 0x7f41880d06c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:57552/0 (socket says 172.21.15.186:57552) 2024-03-28T15:10:47.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.936+0000 7f417effd640 1 --2- 172.21.15.186:0/1249731925 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f418815e850 0x7f41880cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.937+0000 7f4187fff640 1 -- 172.21.15.186:0/1249731925 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f418815e850 msgr2=0x7f41880cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:47.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.937+0000 7f4187fff640 1 --2- 172.21.15.186:0/1249731925 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f418815e850 0x7f41880cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.937+0000 7f4186ffd640 1 -- 172.21.15.186:0/1249731925 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2331333040 0 0) 0x7f41880cac20 con 0x7f4188154b10 2024-03-28T15:10:47.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.937+0000 7f4187fff640 1 -- 172.21.15.186:0/1249731925 >> v1:172.21.15.191:6789/0 conn(0x7f4188154b10 legacy=0x7f41880d06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.937+0000 7f4187fff640 1 -- 172.21.15.186:0/1249731925 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41880caf00 con 0x7f41880cc180 2024-03-28T15:10:47.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.937+0000 7f4187fff640 1 --2- 172.21.15.186:0/1249731925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41880cc180 0x7f41880c5b60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4180009e50 tx=0x7f418002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fce5a3c29fc64a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:47.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.938+0000 7f4186ffd640 1 -- 172.21.15.186:0/1249731925 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418002cb00 con 0x7f41880cc180 2024-03-28T15:10:47.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.938+0000 7f4186ffd640 1 -- 172.21.15.186:0/1249731925 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f418002cca0 con 0x7f41880cc180 2024-03-28T15:10:47.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.938+0000 7f4186ffd640 1 -- 172.21.15.186:0/1249731925 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180034610 con 0x7f41880cc180 2024-03-28T15:10:47.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.938+0000 7f418e08a640 1 -- 172.21.15.186:0/1249731925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41880cc180 msgr2=0x7f41880c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.938+0000 7f418e08a640 1 --2- 172.21.15.186:0/1249731925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41880cc180 0x7f41880c5b60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4180009e50 tx=0x7f418002bf60 comp rx=0 tx=0).stop 2024-03-28T15:10:47.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.938+0000 7f418e08a640 1 -- 172.21.15.186:0/1249731925 shutdown_connections 2024-03-28T15:10:47.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.938+0000 7f418e08a640 1 --2- 172.21.15.186:0/1249731925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41880cc180 0x7f41880c5b60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.938+0000 7f418e08a640 1 --2- 172.21.15.186:0/1249731925 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f418815e850 0x7f41880cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.938+0000 7f418e08a640 1 -- 172.21.15.186:0/1249731925 >> 172.21.15.186:0/1249731925 conn(0x7f41880733a0 msgr2=0x7f41880737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:47.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.938+0000 7f418e08a640 1 -- 172.21.15.186:0/1249731925 shutdown_connections 2024-03-28T15:10:47.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.938+0000 7f418e08a640 1 -- 172.21.15.186:0/1249731925 wait complete. 2024-03-28T15:10:47.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.939+0000 7f418e08a640 1 Processor -- start 2024-03-28T15:10:47.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.939+0000 7f418e08a640 1 -- start start 2024-03-28T15:10:47.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f418e08a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41880cc180 0x7f4188156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f418e08a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4188154b10 0x7f418815d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.946 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f418e08a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f418815e850 0x7f41880cf360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f418e08a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41880c5110 con 0x7f41880cc180 2024-03-28T15:10:47.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f418e08a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f418815edb0 con 0x7f4188154b10 2024-03-28T15:10:47.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f418e08a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f41880c56d0 con 0x7f418815e850 2024-03-28T15:10:47.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f41877fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41880cc180 0x7f4188156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f41877fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41880cc180 0x7f4188156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36716/0 (socket says 172.21.15.186:36716) 2024-03-28T15:10:47.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f41877fe640 1 -- 172.21.15.186:0/2247548228 learned_addr learned my addr 172.21.15.186:0/2247548228 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:47.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f4187fff640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f418815e850 0x7f41880cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f417effd640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4188154b10 0x7f418815d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f41877fe640 1 -- 172.21.15.186:0/2247548228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f418815e850 msgr2=0x7f41880cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f41877fe640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f418815e850 0x7f41880cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f41877fe640 1 -- 172.21.15.186:0/2247548228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4188154b10 msgr2=0x7f418815d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f41877fe640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4188154b10 0x7f418815d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.940+0000 7f41877fe640 1 -- 172.21.15.186:0/2247548228 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41880c7690 con 0x7f41880cc180 2024-03-28T15:10:47.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.941+0000 7f41877fe640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41880cc180 0x7f4188156170 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f417800c500 tx=0x7f417800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:47.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.941+0000 7f4184ff9640 1 -- 172.21.15.186:0/2247548228 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178018070 con 0x7f41880cc180 2024-03-28T15:10:47.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.941+0000 7f4184ff9640 1 -- 172.21.15.186:0/2247548228 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4178007580 con 0x7f41880cc180 2024-03-28T15:10:47.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.941+0000 7f418e08a640 1 -- 172.21.15.186:0/2247548228 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41880c7930 con 0x7f41880cc180 2024-03-28T15:10:47.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.941+0000 7f4184ff9640 1 -- 172.21.15.186:0/2247548228 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178004a20 con 0x7f41880cc180 2024-03-28T15:10:47.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.941+0000 7f418e08a640 1 -- 172.21.15.186:0/2247548228 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f418815d0f0 con 0x7f41880cc180 2024-03-28T15:10:47.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.942+0000 7f418e08a640 1 -- 172.21.15.186:0/2247548228 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4188069570 con 0x7f41880cc180 2024-03-28T15:10:47.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.942+0000 7f4184ff9640 1 -- 172.21.15.186:0/2247548228 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f417801b070 con 0x7f41880cc180 2024-03-28T15:10:47.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.942+0000 7f4184ff9640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f416806c560 0x7f416806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.943+0000 7f4184ff9640 1 -- 172.21.15.186:0/2247548228 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f417808aac0 con 0x7f41880cc180 2024-03-28T15:10:47.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.943+0000 7f417effd640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f416806c560 0x7f416806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.943+0000 7f4184ff9640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] conn(0x7f4168074c00 0x7f4168077080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:47.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.943+0000 7f4184ff9640 1 -- 172.21.15.186:0/2247548228 --> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4168077750 con 0x7f4168074c00 2024-03-28T15:10:47.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.943+0000 7f4184ff9640 1 -- 172.21.15.186:0/2247548228 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f417808f2c0 con 0x7f41880cc180 2024-03-28T15:10:47.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.943+0000 7f4187fff640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] conn(0x7f4168074c00 0x7f4168077080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:47.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.943+0000 7f417effd640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f416806c560 0x7f416806ea20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f418815bbf0 tx=0x7f4170009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:47.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.943+0000 7f4187fff640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] conn(0x7f4168074c00 0x7f4168077080 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:47.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.944+0000 7f4184ff9640 1 -- 172.21.15.186:0/2247548228 <== osd.1 v2:172.21.15.186:6800/2980378029 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4168077750 con 0x7f4168074c00 2024-03-28T15:10:47.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f418e08a640 1 -- 172.21.15.186:0/2247548228 --> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f41880c7b00 con 0x7f4168074c00 2024-03-28T15:10:47.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f4184ff9640 1 -- 172.21.15.186:0/2247548228 <== osd.1 v2:172.21.15.186:6800/2980378029 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f41880c7b00 con 0x7f4168074c00 2024-03-28T15:10:47.958 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-28T15:10:47.958 INFO:teuthology.orchestra.run.smithi186.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T15:10:47.958 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-28T15:10:47.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f418e08a640 1 -- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] conn(0x7f4168074c00 msgr2=0x7f4168077080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f418e08a640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:6800/2980378029,v1:172.21.15.186:6801/2980378029] conn(0x7f4168074c00 0x7f4168077080 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f418e08a640 1 -- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f416806c560 msgr2=0x7f416806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f418e08a640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f416806c560 0x7f416806ea20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f418815bbf0 tx=0x7f4170009190 comp rx=0 tx=0).stop 2024-03-28T15:10:47.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f418e08a640 1 -- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41880cc180 msgr2=0x7f4188156170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:47.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f418e08a640 1 --2- 172.21.15.186:0/2247548228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41880cc180 0x7f4188156170 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f417800c500 tx=0x7f417800c9c0 comp rx=0 tx=0).stop 2024-03-28T15:10:47.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f4187fff640 1 -- 172.21.15.186:0/2247548228 reap_dead start 2024-03-28T15:10:47.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f418e08a640 1 -- 172.21.15.186:0/2247548228 shutdown_connections 2024-03-28T15:10:47.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f418e08a640 1 -- 172.21.15.186:0/2247548228 >> 172.21.15.186:0/2247548228 conn(0x7f41880733a0 msgr2=0x7f418814f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:47.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f418e08a640 1 -- 172.21.15.186:0/2247548228 shutdown_connections 2024-03-28T15:10:47.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:47.957+0000 7f418e08a640 1 -- 172.21.15.186:0/2247548228 wait complete. 2024-03-28T15:10:47.969 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-28T15:10:48.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.071+0000 7f4115c81640 1 Processor -- start 2024-03-28T15:10:48.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.071+0000 7f4115c81640 1 -- start start 2024-03-28T15:10:48.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f4115c81640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f41100c2800 0x7f41100c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f4115c81640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f411006c940 0x7f41100c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f4115c81640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41100c7490 con 0x7f4110150fd0 2024-03-28T15:10:48.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f4115c81640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f41100c1760 con 0x7f411006c940 2024-03-28T15:10:48.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f4115c81640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f41100c1a40 con 0x7f41100c2800 2024-03-28T15:10:48.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f410ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f411006c940 0x7f41100c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f410effd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f41100c2800 0x7f41100c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f410ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f411006c940 0x7f41100c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36732/0 (socket says 172.21.15.186:36732) 2024-03-28T15:10:48.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f410ffff640 1 -- 172.21.15.186:0/4170536359 learned_addr learned my addr 172.21.15.186:0/4170536359 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:48.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f410f7fe640 1 --1- 172.21.15.186:0/4170536359 >> v1:172.21.15.191:6789/0 conn(0x7f4110150fd0 0x7f411015ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:57558/0 (socket says 172.21.15.186:57558) 2024-03-28T15:10:48.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f410effd640 1 -- 172.21.15.186:0/4170536359 >> v2:172.21.15.186:3300/0 conn(0x7f411006c940 msgr2=0x7f41100c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:48.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f410effd640 1 --2- 172.21.15.186:0/4170536359 >> v2:172.21.15.186:3300/0 conn(0x7f411006c940 0x7f41100c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f410effd640 1 -- 172.21.15.186:0/4170536359 >> v1:172.21.15.191:6789/0 conn(0x7f4110150fd0 legacy=0x7f411015ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f410effd640 1 -- 172.21.15.186:0/4170536359 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41100c1d20 con 0x7f41100c2800 2024-03-28T15:10:48.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f410ffff640 1 --2- 172.21.15.186:0/4170536359 >> v2:172.21.15.186:3300/0 conn(0x7f411006c940 0x7f41100c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:48.077 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.072+0000 7f410e7fc640 1 -- 172.21.15.186:0/4170536359 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 202964958 0 0) 0x7f41100c7490 con 0x7f4110150fd0 2024-03-28T15:10:48.077 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.073+0000 7f410effd640 1 --2- 172.21.15.186:0/4170536359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41100c2800 0x7f41100c6d90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4110071990 tx=0x7f410402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf05181df2d3b783 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.077 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.073+0000 7f410e7fc640 1 -- 172.21.15.186:0/4170536359 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104035020 con 0x7f41100c2800 2024-03-28T15:10:48.077 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.073+0000 7f410e7fc640 1 -- 172.21.15.186:0/4170536359 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4104031830 con 0x7f41100c2800 2024-03-28T15:10:48.077 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.073+0000 7f410e7fc640 1 -- 172.21.15.186:0/4170536359 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104031c00 con 0x7f41100c2800 2024-03-28T15:10:48.077 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.074+0000 7f4115c81640 1 -- 172.21.15.186:0/4170536359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41100c2800 msgr2=0x7f41100c6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.077 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.074+0000 7f4115c81640 1 --2- 172.21.15.186:0/4170536359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41100c2800 0x7f41100c6d90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4110071990 tx=0x7f410402b8d0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.077 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.074+0000 7f4115c81640 1 -- 172.21.15.186:0/4170536359 shutdown_connections 2024-03-28T15:10:48.078 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.074+0000 7f4115c81640 1 --2- 172.21.15.186:0/4170536359 >> v2:172.21.15.186:3300/0 conn(0x7f411006c940 0x7f41100c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.078 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.074+0000 7f4115c81640 1 --2- 172.21.15.186:0/4170536359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41100c2800 0x7f41100c6d90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.078 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.074+0000 7f4115c81640 1 -- 172.21.15.186:0/4170536359 >> 172.21.15.186:0/4170536359 conn(0x7f411006c010 msgr2=0x7f41100cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:48.078 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.074+0000 7f4115c81640 1 -- 172.21.15.186:0/4170536359 shutdown_connections 2024-03-28T15:10:48.078 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.074+0000 7f4115c81640 1 -- 172.21.15.186:0/4170536359 wait complete. 2024-03-28T15:10:48.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.075+0000 7f4115c81640 1 Processor -- start 2024-03-28T15:10:48.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.075+0000 7f4115c81640 1 -- start start 2024-03-28T15:10:48.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f4115c81640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411006c940 0x7f41100c4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f4115c81640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41100c2800 0x7f4110156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f4115c81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4110150fd0 0x7f41101f1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f4115c81640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41101513b0 con 0x7f411006c940 2024-03-28T15:10:48.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f4115c81640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f41100c22e0 con 0x7f41100c2800 2024-03-28T15:10:48.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f4115c81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f41100c8790 con 0x7f4110150fd0 2024-03-28T15:10:48.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f410f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411006c940 0x7f41100c4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f410f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411006c940 0x7f41100c4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36738/0 (socket says 172.21.15.186:36738) 2024-03-28T15:10:48.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f410f7fe640 1 -- 172.21.15.186:0/3773311563 learned_addr learned my addr 172.21.15.186:0/3773311563 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:48.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f410effd640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41100c2800 0x7f4110156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f410ffff640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4110150fd0 0x7f41101f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f410f7fe640 1 -- 172.21.15.186:0/3773311563 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4110150fd0 msgr2=0x7f41101f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f410f7fe640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4110150fd0 0x7f41101f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f410f7fe640 1 -- 172.21.15.186:0/3773311563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41100c2800 msgr2=0x7f4110156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f410f7fe640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41100c2800 0x7f4110156120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f410f7fe640 1 -- 172.21.15.186:0/3773311563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41100c5510 con 0x7f411006c940 2024-03-28T15:10:48.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f410f7fe640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411006c940 0x7f41100c4890 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f410000c430 tx=0x7f410000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.076+0000 7f40f7fff640 1 -- 172.21.15.186:0/3773311563 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41000074e0 con 0x7f411006c940 2024-03-28T15:10:48.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.077+0000 7f40f7fff640 1 -- 172.21.15.186:0/3773311563 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41000047e0 con 0x7f411006c940 2024-03-28T15:10:48.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.077+0000 7f4115c81640 1 -- 172.21.15.186:0/3773311563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41100c57d0 con 0x7f411006c940 2024-03-28T15:10:48.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.077+0000 7f40f7fff640 1 -- 172.21.15.186:0/3773311563 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4100004bd0 con 0x7f411006c940 2024-03-28T15:10:48.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.077+0000 7f4115c81640 1 -- 172.21.15.186:0/3773311563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41100c3db0 con 0x7f411006c940 2024-03-28T15:10:48.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.078+0000 7f4115c81640 1 -- 172.21.15.186:0/3773311563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4110069570 con 0x7f411006c940 2024-03-28T15:10:48.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.078+0000 7f40f7fff640 1 -- 172.21.15.186:0/3773311563 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4100011070 con 0x7f411006c940 2024-03-28T15:10:48.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.078+0000 7f40f7fff640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f40fc06c560 0x7f40fc06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.078+0000 7f40f7fff640 1 -- 172.21.15.186:0/3773311563 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f410008b2b0 con 0x7f411006c940 2024-03-28T15:10:48.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.078+0000 7f410effd640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f40fc06c560 0x7f40fc06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.079+0000 7f410effd640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f40fc06c560 0x7f40fc06ea20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4104036000 tx=0x7f4104008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.083+0000 7f40f7fff640 1 -- 172.21.15.186:0/3773311563 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f41000590f0 con 0x7f411006c940 2024-03-28T15:10:48.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.259+0000 7f4115c81640 1 -- 172.21.15.186:0/3773311563 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f4110071520 con 0x7f40fc06c560 2024-03-28T15:10:48.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.260+0000 7f40f7fff640 1 -- 172.21.15.186:0/3773311563 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f4110071520 con 0x7f40fc06c560 2024-03-28T15:10:48.260 INFO:teuthology.orchestra.run.smithi186.stderr:instructed osd(s) 1 to deep-scrub 2024-03-28T15:10:48.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 -- 172.21.15.186:0/3773311563 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f40fc06c560 msgr2=0x7f40fc06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f40fc06c560 0x7f40fc06ea20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4104036000 tx=0x7f4104008030 comp rx=0 tx=0).stop 2024-03-28T15:10:48.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 -- 172.21.15.186:0/3773311563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411006c940 msgr2=0x7f41100c4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411006c940 0x7f41100c4890 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f410000c430 tx=0x7f410000c8f0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 -- 172.21.15.186:0/3773311563 shutdown_connections 2024-03-28T15:10:48.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f40fc06c560 0x7f40fc06ea20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4110150fd0 0x7f41101f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41100c2800 0x7f4110156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 --2- 172.21.15.186:0/3773311563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f411006c940 0x7f41100c4890 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 -- 172.21.15.186:0/3773311563 >> 172.21.15.186:0/3773311563 conn(0x7f411006c010 msgr2=0x7f41101501f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:48.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 -- 172.21.15.186:0/3773311563 shutdown_connections 2024-03-28T15:10:48.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.262+0000 7f4115c81640 1 -- 172.21.15.186:0/3773311563 wait complete. 2024-03-28T15:10:48.274 INFO:tasks.ceph:Scrubbing osd.2 2024-03-28T15:10:48.274 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-28T15:10:48.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.378+0000 7f487cb1e640 1 Processor -- start 2024-03-28T15:10:48.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.378+0000 7f487cb1e640 1 -- start start 2024-03-28T15:10:48.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.379+0000 7f487cb1e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f487815a9c0 0x7f487806cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.379+0000 7f487cb1e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f487806b460 0x7f48780c9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.379+0000 7f487cb1e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48781583e0 con 0x7f4878158000 2024-03-28T15:10:48.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.379+0000 7f487cb1e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f487806d430 con 0x7f487806b460 2024-03-28T15:10:48.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.379+0000 7f487cb1e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f487815ae50 con 0x7f487815a9c0 2024-03-28T15:10:48.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.379+0000 7f4876d9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f487806b460 0x7f48780c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.379+0000 7f4876d9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f487806b460 0x7f48780c9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36754/0 (socket says 172.21.15.186:36754) 2024-03-28T15:10:48.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.379+0000 7f4876d9d640 1 -- 172.21.15.186:0/3710359025 learned_addr learned my addr 172.21.15.186:0/3710359025 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:48.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.379+0000 7f4875d9b640 1 --2- 172.21.15.186:0/3710359025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f487815a9c0 0x7f487806cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.380+0000 7f4876d9d640 1 -- 172.21.15.186:0/3710359025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f487815a9c0 msgr2=0x7f487806cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:48.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.380+0000 7f4876d9d640 1 --2- 172.21.15.186:0/3710359025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f487815a9c0 0x7f487806cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.380+0000 7f487559a640 1 -- 172.21.15.186:0/3710359025 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 739266344 0 0) 0x7f48781583e0 con 0x7f4878158000 2024-03-28T15:10:48.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.380+0000 7f4876d9d640 1 -- 172.21.15.186:0/3710359025 >> v1:172.21.15.191:6789/0 conn(0x7f4878158000 legacy=0x7f48780ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.382 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.380+0000 7f4876d9d640 1 -- 172.21.15.186:0/3710359025 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487815b130 con 0x7f487806b460 2024-03-28T15:10:48.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.380+0000 7f4876d9d640 1 --2- 172.21.15.186:0/3710359025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487806b460 0x7f48780c9250 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4870009870 tx=0x7f487002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8f221633d2d602 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.380+0000 7f487559a640 1 -- 172.21.15.186:0/3710359025 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487002f840 con 0x7f487806b460 2024-03-28T15:10:48.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.380+0000 7f487559a640 1 -- 172.21.15.186:0/3710359025 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f487002f9e0 con 0x7f487806b460 2024-03-28T15:10:48.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.380+0000 7f487559a640 1 -- 172.21.15.186:0/3710359025 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487002fdb0 con 0x7f487806b460 2024-03-28T15:10:48.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.381+0000 7f487cb1e640 1 -- 172.21.15.186:0/3710359025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487806b460 msgr2=0x7f48780c9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.381+0000 7f487cb1e640 1 --2- 172.21.15.186:0/3710359025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487806b460 0x7f48780c9250 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4870009870 tx=0x7f487002ba20 comp rx=0 tx=0).stop 2024-03-28T15:10:48.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.381+0000 7f487cb1e640 1 -- 172.21.15.186:0/3710359025 shutdown_connections 2024-03-28T15:10:48.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.381+0000 7f487cb1e640 1 --2- 172.21.15.186:0/3710359025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487806b460 0x7f48780c9250 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.381+0000 7f487cb1e640 1 --2- 172.21.15.186:0/3710359025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f487815a9c0 0x7f487806cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.381+0000 7f487cb1e640 1 -- 172.21.15.186:0/3710359025 >> 172.21.15.186:0/3710359025 conn(0x7f4878073aa0 msgr2=0x7f487806d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:48.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.381+0000 7f487cb1e640 1 -- 172.21.15.186:0/3710359025 shutdown_connections 2024-03-28T15:10:48.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.381+0000 7f487cb1e640 1 -- 172.21.15.186:0/3710359025 wait complete. 2024-03-28T15:10:48.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.382+0000 7f487cb1e640 1 Processor -- start 2024-03-28T15:10:48.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.382+0000 7f487cb1e640 1 -- start start 2024-03-28T15:10:48.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f487cb1e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f487806b460 0x7f48781565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f487cb1e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4878158000 0x7f4878156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f487cb1e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48780ce350 0x7f487814e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f487cb1e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48780c8d30 con 0x7f48780ce350 2024-03-28T15:10:48.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f487cb1e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f487815b6f0 con 0x7f487806b460 2024-03-28T15:10:48.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f487cb1e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f48780ce830 con 0x7f4878158000 2024-03-28T15:10:48.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f4876d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48780ce350 0x7f487814e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f4876d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48780ce350 0x7f487814e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36770/0 (socket says 172.21.15.186:36770) 2024-03-28T15:10:48.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f4876d9d640 1 -- 172.21.15.186:0/2122403456 learned_addr learned my addr 172.21.15.186:0/2122403456 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:48.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f4875d9b640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4878158000 0x7f4878156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f487659c640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f487806b460 0x7f48781565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f4876d9d640 1 -- 172.21.15.186:0/2122403456 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4878158000 msgr2=0x7f4878156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.383+0000 7f4876d9d640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4878158000 0x7f4878156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.384+0000 7f4876d9d640 1 -- 172.21.15.186:0/2122403456 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f487806b460 msgr2=0x7f48781565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.384+0000 7f4876d9d640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f487806b460 0x7f48781565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.384+0000 7f4876d9d640 1 -- 172.21.15.186:0/2122403456 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4878155040 con 0x7f48780ce350 2024-03-28T15:10:48.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.384+0000 7f4875d9b640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4878158000 0x7f4878156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:48.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.384+0000 7f4876d9d640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48780ce350 0x7f487814e960 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4870009870 tx=0x7f4870004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.384+0000 7f4866ffd640 1 -- 172.21.15.186:0/2122403456 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4870047070 con 0x7f48780ce350 2024-03-28T15:10:48.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.384+0000 7f4866ffd640 1 -- 172.21.15.186:0/2122403456 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4870004230 con 0x7f48780ce350 2024-03-28T15:10:48.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.384+0000 7f487cb1e640 1 -- 172.21.15.186:0/2122403456 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48780cd030 con 0x7f48780ce350 2024-03-28T15:10:48.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.384+0000 7f4866ffd640 1 -- 172.21.15.186:0/2122403456 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4870037940 con 0x7f48780ce350 2024-03-28T15:10:48.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.384+0000 7f487cb1e640 1 -- 172.21.15.186:0/2122403456 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48780cd5f0 con 0x7f48780ce350 2024-03-28T15:10:48.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.385+0000 7f487cb1e640 1 -- 172.21.15.186:0/2122403456 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f48780686d0 con 0x7f48780ce350 2024-03-28T15:10:48.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.385+0000 7f4866ffd640 1 -- 172.21.15.186:0/2122403456 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f487004a070 con 0x7f48780ce350 2024-03-28T15:10:48.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.386+0000 7f4866ffd640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f485806c560 0x7f485806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.386+0000 7f4866ffd640 1 -- 172.21.15.186:0/2122403456 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f48700ad6b0 con 0x7f48780ce350 2024-03-28T15:10:48.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.386+0000 7f487659c640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f485806c560 0x7f485806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.386+0000 7f4866ffd640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] conn(0x7f4858074c10 0x7f4858077090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.386+0000 7f4866ffd640 1 -- 172.21.15.186:0/2122403456 --> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4870045e70 con 0x7f4858074c10 2024-03-28T15:10:48.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.386+0000 7f4866ffd640 1 -- 172.21.15.186:0/2122403456 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f48700b4050 con 0x7f48780ce350 2024-03-28T15:10:48.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.386+0000 7f4875d9b640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] conn(0x7f4858074c10 0x7f4858077090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.386+0000 7f487659c640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f485806c560 0x7f485806ea20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f48680012e0 tx=0x7f4868009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.386+0000 7f4875d9b640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] conn(0x7f4858074c10 0x7f4858077090 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.387+0000 7f4866ffd640 1 -- 172.21.15.186:0/2122403456 <== osd.2 v2:172.21.15.186:6808/2659582963 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4870045e70 con 0x7f4858074c10 2024-03-28T15:10:48.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.400+0000 7f487cb1e640 1 -- 172.21.15.186:0/2122403456 --> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4878068de0 con 0x7f4858074c10 2024-03-28T15:10:48.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.400+0000 7f4866ffd640 1 -- 172.21.15.186:0/2122403456 <== osd.2 v2:172.21.15.186:6808/2659582963 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4878068de0 con 0x7f4858074c10 2024-03-28T15:10:48.401 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-28T15:10:48.401 INFO:teuthology.orchestra.run.smithi186.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T15:10:48.401 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-28T15:10:48.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.400+0000 7f487cb1e640 1 -- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] conn(0x7f4858074c10 msgr2=0x7f4858077090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.400+0000 7f487cb1e640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:6808/2659582963,v1:172.21.15.186:6809/2659582963] conn(0x7f4858074c10 0x7f4858077090 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.401+0000 7f487cb1e640 1 -- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f485806c560 msgr2=0x7f485806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.401+0000 7f487cb1e640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f485806c560 0x7f485806ea20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f48680012e0 tx=0x7f4868009040 comp rx=0 tx=0).stop 2024-03-28T15:10:48.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.401+0000 7f487cb1e640 1 -- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48780ce350 msgr2=0x7f487814e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.401+0000 7f487cb1e640 1 --2- 172.21.15.186:0/2122403456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48780ce350 0x7f487814e960 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4870009870 tx=0x7f4870004730 comp rx=0 tx=0).stop 2024-03-28T15:10:48.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.401+0000 7f4876d9d640 1 -- 172.21.15.186:0/2122403456 reap_dead start 2024-03-28T15:10:48.404 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.401+0000 7f487cb1e640 1 -- 172.21.15.186:0/2122403456 shutdown_connections 2024-03-28T15:10:48.404 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.401+0000 7f487cb1e640 1 -- 172.21.15.186:0/2122403456 >> 172.21.15.186:0/2122403456 conn(0x7f4878073aa0 msgr2=0x7f4878158bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:48.404 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.401+0000 7f487cb1e640 1 -- 172.21.15.186:0/2122403456 shutdown_connections 2024-03-28T15:10:48.404 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.401+0000 7f487cb1e640 1 -- 172.21.15.186:0/2122403456 wait complete. 2024-03-28T15:10:48.412 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-28T15:10:48.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.513+0000 7f61dff81640 1 Processor -- start 2024-03-28T15:10:48.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.513+0000 7f61dff81640 1 -- start start 2024-03-28T15:10:48.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61dff81640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f61d80c8110 0x7f61d806ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61dff81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d806f120 0x7f61d806db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61dff81640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61d80c8600 con 0x7f61d8151870 2024-03-28T15:10:48.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61dff81640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f61d8158070 con 0x7f61d80c8110 2024-03-28T15:10:48.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61dff81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f61d8157740 con 0x7f61d806f120 2024-03-28T15:10:48.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61de51e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d806f120 0x7f61d806db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61dd51c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f61d80c8110 0x7f61d806ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61de51e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d806f120 0x7f61d806db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:56310/0 (socket says 172.21.15.186:56310) 2024-03-28T15:10:48.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61de51e640 1 -- 172.21.15.186:0/617610546 learned_addr learned my addr 172.21.15.186:0/617610546 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:48.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61ddd1d640 1 --1- 172.21.15.186:0/617610546 >> v1:172.21.15.191:6789/0 conn(0x7f61d8151870 0x7f61d80c7390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:57572/0 (socket says 172.21.15.186:57572) 2024-03-28T15:10:48.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61de51e640 1 -- 172.21.15.186:0/617610546 >> v2:172.21.15.191:3300/0 conn(0x7f61d80c8110 msgr2=0x7f61d806ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:48.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61de51e640 1 --2- 172.21.15.186:0/617610546 >> v2:172.21.15.191:3300/0 conn(0x7f61d80c8110 0x7f61d806ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61de51e640 1 -- 172.21.15.186:0/617610546 >> v1:172.21.15.191:6789/0 conn(0x7f61d8151870 legacy=0x7f61d80c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61de51e640 1 -- 172.21.15.186:0/617610546 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d8157a20 con 0x7f61d806f120 2024-03-28T15:10:48.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.514+0000 7f61dd51c640 1 --2- 172.21.15.186:0/617610546 >> v2:172.21.15.191:3300/0 conn(0x7f61d80c8110 0x7f61d806ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:48.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.515+0000 7f61de51e640 1 --2- 172.21.15.186:0/617610546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d806f120 0x7f61d806db50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009e50 tx=0x7f61d002bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26dd2024635e6191 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.515+0000 7f61dcd1b640 1 -- 172.21.15.186:0/617610546 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d002cb90 con 0x7f61d806f120 2024-03-28T15:10:48.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.515+0000 7f61dcd1b640 1 -- 172.21.15.186:0/617610546 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61d002cd30 con 0x7f61d806f120 2024-03-28T15:10:48.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.515+0000 7f61dcd1b640 1 -- 172.21.15.186:0/617610546 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d0034610 con 0x7f61d806f120 2024-03-28T15:10:48.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.516+0000 7f61dff81640 1 -- 172.21.15.186:0/617610546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d806f120 msgr2=0x7f61d806db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.516+0000 7f61dff81640 1 --2- 172.21.15.186:0/617610546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d806f120 0x7f61d806db50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009e50 tx=0x7f61d002bff0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.516+0000 7f61dff81640 1 -- 172.21.15.186:0/617610546 shutdown_connections 2024-03-28T15:10:48.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.516+0000 7f61dff81640 1 --2- 172.21.15.186:0/617610546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d806f120 0x7f61d806db50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.516+0000 7f61dff81640 1 --2- 172.21.15.186:0/617610546 >> v2:172.21.15.191:3300/0 conn(0x7f61d80c8110 0x7f61d806ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.516+0000 7f61dff81640 1 -- 172.21.15.186:0/617610546 >> 172.21.15.186:0/617610546 conn(0x7f61d806c010 msgr2=0x7f61d80cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:48.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.516+0000 7f61dff81640 1 -- 172.21.15.186:0/617610546 shutdown_connections 2024-03-28T15:10:48.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.516+0000 7f61dff81640 1 -- 172.21.15.186:0/617610546 wait complete. 2024-03-28T15:10:48.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.517+0000 7f61dff81640 1 Processor -- start 2024-03-28T15:10:48.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.517+0000 7f61dff81640 1 -- start start 2024-03-28T15:10:48.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dff81640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61d806f120 0x7f61d81de530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dff81640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61d80c8110 0x7f61d81ef320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dff81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d8151870 0x7f61d81f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dff81640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f61d8073e10 con 0x7f61d806f120 2024-03-28T15:10:48.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dff81640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f61d8151c50 con 0x7f61d80c8110 2024-03-28T15:10:48.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dff81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f61d8073f90 con 0x7f61d8151870 2024-03-28T15:10:48.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61ddd1d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61d806f120 0x7f61d81de530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61ddd1d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61d806f120 0x7f61d81de530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36782/0 (socket says 172.21.15.186:36782) 2024-03-28T15:10:48.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61ddd1d640 1 -- 172.21.15.186:0/3963771782 learned_addr learned my addr 172.21.15.186:0/3963771782 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:48.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dd51c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61d80c8110 0x7f61d81ef320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.523 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61de51e640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d8151870 0x7f61d81f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dd51c640 1 -- 172.21.15.186:0/3963771782 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d8151870 msgr2=0x7f61d81f1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dd51c640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d8151870 0x7f61d81f1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dd51c640 1 -- 172.21.15.186:0/3963771782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61d806f120 msgr2=0x7f61d81de530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dd51c640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61d806f120 0x7f61d81de530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dd51c640 1 -- 172.21.15.186:0/3963771782 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d81f2310 con 0x7f61d80c8110 2024-03-28T15:10:48.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61ddd1d640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61d806f120 0x7f61d81de530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:48.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dd51c640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61d80c8110 0x7f61d81ef320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f61c800e930 tx=0x7f61c800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.524 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61ce7fc640 1 -- 172.21.15.186:0/3963771782 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c8017070 con 0x7f61d80c8110 2024-03-28T15:10:48.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61ce7fc640 1 -- 172.21.15.186:0/3963771782 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c800ca00 con 0x7f61d80c8110 2024-03-28T15:10:48.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dff81640 1 -- 172.21.15.186:0/3963771782 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61d81ee860 con 0x7f61d80c8110 2024-03-28T15:10:48.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61ce7fc640 1 -- 172.21.15.186:0/3963771782 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c800cdf0 con 0x7f61d80c8110 2024-03-28T15:10:48.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.518+0000 7f61dff81640 1 -- 172.21.15.186:0/3963771782 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61d81f2880 con 0x7f61d80c8110 2024-03-28T15:10:48.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.519+0000 7f61dff81640 1 -- 172.21.15.186:0/3963771782 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61d80c7bc0 con 0x7f61d80c8110 2024-03-28T15:10:48.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.520+0000 7f61ce7fc640 1 -- 172.21.15.186:0/3963771782 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f61c801a070 con 0x7f61d80c8110 2024-03-28T15:10:48.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.520+0000 7f61ce7fc640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f61bc06c530 0x7f61bc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.520+0000 7f61ce7fc640 1 -- 172.21.15.186:0/3963771782 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f61c808bab0 con 0x7f61d80c8110 2024-03-28T15:10:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.520+0000 7f61ddd1d640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f61bc06c530 0x7f61bc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.520+0000 7f61ddd1d640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f61bc06c530 0x7f61bc06e9f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f61d4004e00 tx=0x7f61d4007f10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.525+0000 7f61ce7fc640 1 -- 172.21.15.186:0/3963771782 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f61c80597c0 con 0x7f61d80c8110 2024-03-28T15:10:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.699+0000 7f61dff81640 1 -- 172.21.15.186:0/3963771782 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f61d80739e0 con 0x7f61bc06c530 2024-03-28T15:10:48.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.700+0000 7f61ce7fc640 1 -- 172.21.15.186:0/3963771782 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f61d80739e0 con 0x7f61bc06c530 2024-03-28T15:10:48.701 INFO:teuthology.orchestra.run.smithi186.stderr:instructed osd(s) 2 to deep-scrub 2024-03-28T15:10:48.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 -- 172.21.15.186:0/3963771782 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f61bc06c530 msgr2=0x7f61bc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f61bc06c530 0x7f61bc06e9f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f61d4004e00 tx=0x7f61d4007f10 comp rx=0 tx=0).stop 2024-03-28T15:10:48.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 -- 172.21.15.186:0/3963771782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61d80c8110 msgr2=0x7f61d81ef320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61d80c8110 0x7f61d81ef320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f61c800e930 tx=0x7f61c800edf0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 -- 172.21.15.186:0/3963771782 shutdown_connections 2024-03-28T15:10:48.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f61bc06c530 0x7f61bc06e9f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d8151870 0x7f61d81f1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61d80c8110 0x7f61d81ef320 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 --2- 172.21.15.186:0/3963771782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61d806f120 0x7f61d81de530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 -- 172.21.15.186:0/3963771782 >> 172.21.15.186:0/3963771782 conn(0x7f61d806c010 msgr2=0x7f61d80c88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:48.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 -- 172.21.15.186:0/3963771782 shutdown_connections 2024-03-28T15:10:48.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.702+0000 7f61dff81640 1 -- 172.21.15.186:0/3963771782 wait complete. 2024-03-28T15:10:48.714 INFO:tasks.ceph:Scrubbing osd.3 2024-03-28T15:10:48.714 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-28T15:10:48.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.821+0000 7f41c2ec2640 1 Processor -- start 2024-03-28T15:10:48.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.821+0000 7f41c2ec2640 1 -- start start 2024-03-28T15:10:48.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.821+0000 7f41c2ec2640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41bc0cd0c0 0x7f41bc0c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.821+0000 7f41c2ec2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41bc0cb1d0 0x7f41bc0c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.821+0000 7f41c2ec2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41bc0cc360 con 0x7f41bc152960 2024-03-28T15:10:48.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.821+0000 7f41c2ec2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f41bc15c490 con 0x7f41bc0cb1d0 2024-03-28T15:10:48.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.821+0000 7f41c2ec2640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f41bc0ca600 con 0x7f41bc0cd0c0 2024-03-28T15:10:48.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.822+0000 7f41c145f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41bc0cb1d0 0x7f41bc0c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.822+0000 7f41c145f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41bc0cb1d0 0x7f41bc0c62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49616/0 (socket says 172.21.15.186:49616) 2024-03-28T15:10:48.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.822+0000 7f41c145f640 1 -- 172.21.15.186:0/617761811 learned_addr learned my addr 172.21.15.186:0/617761811 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:48.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.822+0000 7f41bbfff640 1 --2- 172.21.15.186:0/617761811 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41bc0cd0c0 0x7f41bc0c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.822+0000 7f41c145f640 1 -- 172.21.15.186:0/617761811 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41bc0cd0c0 msgr2=0x7f41bc0c5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:48.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.822+0000 7f41c145f640 1 --2- 172.21.15.186:0/617761811 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41bc0cd0c0 0x7f41bc0c5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.822+0000 7f41bb7fe640 1 -- 172.21.15.186:0/617761811 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2232368905 0 0) 0x7f41bc0cc360 con 0x7f41bc152960 2024-03-28T15:10:48.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.822+0000 7f41c145f640 1 -- 172.21.15.186:0/617761811 >> v1:172.21.15.191:6789/0 conn(0x7f41bc152960 legacy=0x7f41bc15bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.822+0000 7f41c145f640 1 -- 172.21.15.186:0/617761811 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41bc071d00 con 0x7f41bc0cb1d0 2024-03-28T15:10:48.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41c145f640 1 --2- 172.21.15.186:0/617761811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41bc0cb1d0 0x7f41bc0c62b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f41b4009e50 tx=0x7f41b402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902f3c4359805574 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41bb7fe640 1 -- 172.21.15.186:0/617761811 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b402cb00 con 0x7f41bc0cb1d0 2024-03-28T15:10:48.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41bb7fe640 1 -- 172.21.15.186:0/617761811 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41b402cca0 con 0x7f41bc0cb1d0 2024-03-28T15:10:48.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41bb7fe640 1 -- 172.21.15.186:0/617761811 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b4034610 con 0x7f41bc0cb1d0 2024-03-28T15:10:48.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41c2ec2640 1 -- 172.21.15.186:0/617761811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41bc0cb1d0 msgr2=0x7f41bc0c62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41c2ec2640 1 --2- 172.21.15.186:0/617761811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41bc0cb1d0 0x7f41bc0c62b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f41b4009e50 tx=0x7f41b402bf60 comp rx=0 tx=0).stop 2024-03-28T15:10:48.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41c2ec2640 1 -- 172.21.15.186:0/617761811 shutdown_connections 2024-03-28T15:10:48.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41c2ec2640 1 --2- 172.21.15.186:0/617761811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41bc0cb1d0 0x7f41bc0c62b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41c2ec2640 1 --2- 172.21.15.186:0/617761811 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41bc0cd0c0 0x7f41bc0c5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41c2ec2640 1 -- 172.21.15.186:0/617761811 >> 172.21.15.186:0/617761811 conn(0x7f41bc0676a0 msgr2=0x7f41bc06f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:48.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41c2ec2640 1 -- 172.21.15.186:0/617761811 shutdown_connections 2024-03-28T15:10:48.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.823+0000 7f41c2ec2640 1 -- 172.21.15.186:0/617761811 wait complete. 2024-03-28T15:10:48.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.824+0000 7f41c2ec2640 1 Processor -- start 2024-03-28T15:10:48.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.825+0000 7f41c2ec2640 1 -- start start 2024-03-28T15:10:48.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.825+0000 7f41c2ec2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41bc0cb1d0 0x7f41bc145950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.825+0000 7f41c2ec2640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41bc0cd0c0 0x7f41bc15b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.825+0000 7f41c2ec2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41bc152960 0x7f41bc15b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.825+0000 7f41c2ec2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc073930 con 0x7f41bc152960 2024-03-28T15:10:48.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.825+0000 7f41c2ec2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc152d40 con 0x7f41bc0cb1d0 2024-03-28T15:10:48.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.825+0000 7f41c2ec2640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f41bc15be10 con 0x7f41bc0cd0c0 2024-03-28T15:10:48.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.825+0000 7f41c145f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41bc152960 0x7f41bc15b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.825+0000 7f41c145f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41bc152960 0x7f41bc15b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49624/0 (socket says 172.21.15.186:49624) 2024-03-28T15:10:48.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.825+0000 7f41bbfff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41bc0cd0c0 0x7f41bc15b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41c145f640 1 -- 172.21.15.186:0/3665347187 learned_addr learned my addr 172.21.15.186:0/3665347187 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:48.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41c0c5e640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41bc0cb1d0 0x7f41bc145950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41c145f640 1 -- 172.21.15.186:0/3665347187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41bc0cd0c0 msgr2=0x7f41bc15b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41c145f640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41bc0cd0c0 0x7f41bc15b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41c145f640 1 -- 172.21.15.186:0/3665347187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41bc0cb1d0 msgr2=0x7f41bc145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41c145f640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41bc0cb1d0 0x7f41bc145950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41c145f640 1 -- 172.21.15.186:0/3665347187 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41bc15c0b0 con 0x7f41bc152960 2024-03-28T15:10:48.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41c0c5e640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41bc0cb1d0 0x7f41bc145950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:48.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41c145f640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41bc152960 0x7f41bc15b5e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f41b4036040 tx=0x7f41b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41b97fa640 1 -- 172.21.15.186:0/3665347187 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b4043020 con 0x7f41bc152960 2024-03-28T15:10:48.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41b97fa640 1 -- 172.21.15.186:0/3665347187 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41b402ce50 con 0x7f41bc152960 2024-03-28T15:10:48.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41b97fa640 1 -- 172.21.15.186:0/3665347187 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b4009510 con 0x7f41bc152960 2024-03-28T15:10:48.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41c2ec2640 1 -- 172.21.15.186:0/3665347187 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41bc0c6ed0 con 0x7f41bc152960 2024-03-28T15:10:48.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.826+0000 7f41c2ec2640 1 -- 172.21.15.186:0/3665347187 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41bc15a8a0 con 0x7f41bc152960 2024-03-28T15:10:48.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.827+0000 7f41c2ec2640 1 -- 172.21.15.186:0/3665347187 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41bc069570 con 0x7f41bc152960 2024-03-28T15:10:48.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.828+0000 7f41b97fa640 1 -- 172.21.15.186:0/3665347187 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f41b4046020 con 0x7f41bc152960 2024-03-28T15:10:48.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.828+0000 7f41b97fa640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f41a806c530 0x7f41a806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.828+0000 7f41b97fa640 1 -- 172.21.15.186:0/3665347187 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f41b40add30 con 0x7f41bc152960 2024-03-28T15:10:48.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.828+0000 7f41c0c5e640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f41a806c530 0x7f41a806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.828+0000 7f41b97fa640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] conn(0x7f41a8074be0 0x7f41a8077060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.828+0000 7f41b97fa640 1 -- 172.21.15.186:0/3665347187 --> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41b404ee50 con 0x7f41a8074be0 2024-03-28T15:10:48.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.828+0000 7f41b97fa640 1 -- 172.21.15.186:0/3665347187 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f41b40b0430 con 0x7f41bc152960 2024-03-28T15:10:48.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.828+0000 7f41bbfff640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] conn(0x7f41a8074be0 0x7f41a8077060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.828+0000 7f41c0c5e640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f41a806c530 0x7f41a806e9f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f41ac000f60 tx=0x7f41ac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.829+0000 7f41bbfff640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] conn(0x7f41a8074be0 0x7f41a8077060 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.829+0000 7f41b97fa640 1 -- 172.21.15.186:0/3665347187 <== osd.3 v2:172.21.15.186:6816/2463630609 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f41b404ee50 con 0x7f41a8074be0 2024-03-28T15:10:48.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.842+0000 7f41c2ec2640 1 -- 172.21.15.186:0/3665347187 --> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f41bc0cada0 con 0x7f41a8074be0 2024-03-28T15:10:48.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.842+0000 7f41b97fa640 1 -- 172.21.15.186:0/3665347187 <== osd.3 v2:172.21.15.186:6816/2463630609 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f41bc0cada0 con 0x7f41a8074be0 2024-03-28T15:10:48.843 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-28T15:10:48.843 INFO:teuthology.orchestra.run.smithi186.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T15:10:48.843 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-28T15:10:48.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.842+0000 7f41c2ec2640 1 -- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] conn(0x7f41a8074be0 msgr2=0x7f41a8077060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.842+0000 7f41c2ec2640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:6816/2463630609,v1:172.21.15.186:6817/2463630609] conn(0x7f41a8074be0 0x7f41a8077060 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.842+0000 7f41c2ec2640 1 -- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f41a806c530 msgr2=0x7f41a806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.842+0000 7f41c2ec2640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f41a806c530 0x7f41a806e9f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f41ac000f60 tx=0x7f41ac009040 comp rx=0 tx=0).stop 2024-03-28T15:10:48.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.842+0000 7f41c2ec2640 1 -- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41bc152960 msgr2=0x7f41bc15b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.842+0000 7f41c2ec2640 1 --2- 172.21.15.186:0/3665347187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41bc152960 0x7f41bc15b5e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f41b4036040 tx=0x7f41b4004a40 comp rx=0 tx=0).stop 2024-03-28T15:10:48.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.843+0000 7f41c145f640 1 -- 172.21.15.186:0/3665347187 reap_dead start 2024-03-28T15:10:48.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.843+0000 7f41c2ec2640 1 -- 172.21.15.186:0/3665347187 shutdown_connections 2024-03-28T15:10:48.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.843+0000 7f41c2ec2640 1 -- 172.21.15.186:0/3665347187 >> 172.21.15.186:0/3665347187 conn(0x7f41bc0676a0 msgr2=0x7f41bc1513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:48.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.843+0000 7f41c2ec2640 1 -- 172.21.15.186:0/3665347187 shutdown_connections 2024-03-28T15:10:48.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.843+0000 7f41c2ec2640 1 -- 172.21.15.186:0/3665347187 wait complete. 2024-03-28T15:10:48.854 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-28T15:10:48.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.956+0000 7ff2238c0640 1 Processor -- start 2024-03-28T15:10:48.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.956+0000 7ff2238c0640 1 -- start start 2024-03-28T15:10:48.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff2238c0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff21c150fd0 0x7ff21c0c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff2238c0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff21c06c940 0x7ff21c0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff2238c0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff21c0c7490 con 0x7ff21c0c2800 2024-03-28T15:10:48.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff2238c0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7ff21c0c1760 con 0x7ff21c150fd0 2024-03-28T15:10:48.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff2238c0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff21c0c1a40 con 0x7ff21c06c940 2024-03-28T15:10:48.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff220e5b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff21c0c2800 0x7ff21c15ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:60444/0 (socket says 172.21.15.186:60444) 2024-03-28T15:10:48.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff220e5b640 1 -- 172.21.15.186:0/2816219891 learned_addr learned my addr 172.21.15.186:0/2816219891 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:48.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff221e5d640 1 --2- 172.21.15.186:0/2816219891 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff21c06c940 0x7ff21c0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff22165c640 1 --2- 172.21.15.186:0/2816219891 >> v2:172.21.15.191:3300/0 conn(0x7ff21c150fd0 0x7ff21c0c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff213fff640 1 -- 172.21.15.186:0/2816219891 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3760526500 0 0) 0x7ff21c0c7490 con 0x7ff21c0c2800 2024-03-28T15:10:48.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff213fff640 1 -- 172.21.15.186:0/2816219891 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff208003520 con 0x7ff21c0c2800 2024-03-28T15:10:48.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff22165c640 1 -- 172.21.15.186:0/2816219891 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff21c06c940 msgr2=0x7ff21c0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:48.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff22165c640 1 --2- 172.21.15.186:0/2816219891 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff21c06c940 0x7ff21c0c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff22165c640 1 -- 172.21.15.186:0/2816219891 >> v1:172.21.15.186:6789/0 conn(0x7ff21c0c2800 legacy=0x7ff21c15ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff22165c640 1 -- 172.21.15.186:0/2816219891 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21c0c1d20 con 0x7ff21c150fd0 2024-03-28T15:10:48.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.957+0000 7ff221e5d640 1 --2- 172.21.15.186:0/2816219891 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff21c06c940 0x7ff21c0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:48.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.958+0000 7ff22165c640 1 --2- 172.21.15.186:0/2816219891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff21c150fd0 0x7ff21c0c6d90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff2180097a0 tx=0x7ff21802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4754abae72553bb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.958+0000 7ff213fff640 1 -- 172.21.15.186:0/2816219891 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218035020 con 0x7ff21c150fd0 2024-03-28T15:10:48.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.958+0000 7ff213fff640 1 -- 172.21.15.186:0/2816219891 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff218031830 con 0x7ff21c150fd0 2024-03-28T15:10:48.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.958+0000 7ff213fff640 1 -- 172.21.15.186:0/2816219891 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218031be0 con 0x7ff21c150fd0 2024-03-28T15:10:48.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.959+0000 7ff2238c0640 1 -- 172.21.15.186:0/2816219891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff21c150fd0 msgr2=0x7ff21c0c6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.959+0000 7ff2238c0640 1 --2- 172.21.15.186:0/2816219891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff21c150fd0 0x7ff21c0c6d90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff2180097a0 tx=0x7ff21802b8d0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.959+0000 7ff2238c0640 1 -- 172.21.15.186:0/2816219891 shutdown_connections 2024-03-28T15:10:48.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.959+0000 7ff2238c0640 1 --2- 172.21.15.186:0/2816219891 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff21c06c940 0x7ff21c0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.959+0000 7ff2238c0640 1 --2- 172.21.15.186:0/2816219891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff21c150fd0 0x7ff21c0c6d90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.959+0000 7ff2238c0640 1 -- 172.21.15.186:0/2816219891 >> 172.21.15.186:0/2816219891 conn(0x7ff21c06c010 msgr2=0x7ff21c0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:48.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.959+0000 7ff2238c0640 1 -- 172.21.15.186:0/2816219891 shutdown_connections 2024-03-28T15:10:48.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.959+0000 7ff2238c0640 1 -- 172.21.15.186:0/2816219891 wait complete. 2024-03-28T15:10:48.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.960+0000 7ff2238c0640 1 Processor -- start 2024-03-28T15:10:48.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.960+0000 7ff2238c0640 1 -- start start 2024-03-28T15:10:48.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.961+0000 7ff2238c0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff21c06c940 0x7ff21c0c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.961+0000 7ff2238c0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff21c0c2800 0x7ff21c153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.961+0000 7ff2238c0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff21c150fd0 0x7ff21c142b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.961+0000 7ff2238c0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1513b0 con 0x7ff21c0c2800 2024-03-28T15:10:48.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.961+0000 7ff2238c0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c0c22e0 con 0x7ff21c06c940 2024-03-28T15:10:48.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.961+0000 7ff2238c0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff21c0c8790 con 0x7ff21c150fd0 2024-03-28T15:10:48.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.961+0000 7ff220e5b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff21c0c2800 0x7ff21c153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.961+0000 7ff220e5b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff21c0c2800 0x7ff21c153aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49640/0 (socket says 172.21.15.186:49640) 2024-03-28T15:10:48.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.961+0000 7ff220e5b640 1 -- 172.21.15.186:0/1764971930 learned_addr learned my addr 172.21.15.186:0/1764971930 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:48.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.961+0000 7ff221e5d640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff21c150fd0 0x7ff21c142b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.961+0000 7ff22165c640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff21c06c940 0x7ff21c0c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.962+0000 7ff220e5b640 1 -- 172.21.15.186:0/1764971930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff21c150fd0 msgr2=0x7ff21c142b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.962+0000 7ff220e5b640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff21c150fd0 0x7ff21c142b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.962+0000 7ff220e5b640 1 -- 172.21.15.186:0/1764971930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff21c06c940 msgr2=0x7ff21c0c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:48.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.962+0000 7ff220e5b640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff21c06c940 0x7ff21c0c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:48.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.962+0000 7ff220e5b640 1 -- 172.21.15.186:0/1764971930 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21c14a500 con 0x7ff21c0c2800 2024-03-28T15:10:48.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.962+0000 7ff220e5b640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff21c0c2800 0x7ff21c153aa0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00c530 tx=0x7ff20c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.962+0000 7ff211ffb640 1 -- 172.21.15.186:0/1764971930 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c017070 con 0x7ff21c0c2800 2024-03-28T15:10:48.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.962+0000 7ff211ffb640 1 -- 172.21.15.186:0/1764971930 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff20c005ce0 con 0x7ff21c0c2800 2024-03-28T15:10:48.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.962+0000 7ff2238c0640 1 -- 172.21.15.186:0/1764971930 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff21c14a7a0 con 0x7ff21c0c2800 2024-03-28T15:10:48.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.962+0000 7ff211ffb640 1 -- 172.21.15.186:0/1764971930 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0072b0 con 0x7ff21c0c2800 2024-03-28T15:10:48.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.962+0000 7ff2238c0640 1 -- 172.21.15.186:0/1764971930 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff21c155790 con 0x7ff21c0c2800 2024-03-28T15:10:48.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.963+0000 7ff2238c0640 1 -- 172.21.15.186:0/1764971930 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff21c069570 con 0x7ff21c0c2800 2024-03-28T15:10:48.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.963+0000 7ff211ffb640 1 -- 172.21.15.186:0/1764971930 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff20c01a070 con 0x7ff21c0c2800 2024-03-28T15:10:48.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.964+0000 7ff211ffb640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff20006c530 0x7ff20006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:48.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.964+0000 7ff211ffb640 1 -- 172.21.15.186:0/1764971930 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7ff20c08abf0 con 0x7ff21c0c2800 2024-03-28T15:10:48.976 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.964+0000 7ff22165c640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff20006c530 0x7ff20006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:48.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.964+0000 7ff22165c640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff20006c530 0x7ff20006e9f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff218036000 tx=0x7ff218008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:48.977 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:48.968+0000 7ff211ffb640 1 -- 172.21.15.186:0/1764971930 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff20c058af0 con 0x7ff21c0c2800 2024-03-28T15:10:49.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.146+0000 7ff2238c0640 1 -- 172.21.15.186:0/1764971930 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7ff21c0c8570 con 0x7ff20006c530 2024-03-28T15:10:49.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.147+0000 7ff211ffb640 1 -- 172.21.15.186:0/1764971930 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff21c0c8570 con 0x7ff20006c530 2024-03-28T15:10:49.147 INFO:teuthology.orchestra.run.smithi186.stderr:instructed osd(s) 3 to deep-scrub 2024-03-28T15:10:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 -- 172.21.15.186:0/1764971930 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff20006c530 msgr2=0x7ff20006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff20006c530 0x7ff20006e9f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff218036000 tx=0x7ff218008030 comp rx=0 tx=0).stop 2024-03-28T15:10:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 -- 172.21.15.186:0/1764971930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff21c0c2800 msgr2=0x7ff21c153aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff21c0c2800 0x7ff21c153aa0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00c530 tx=0x7ff20c00c9f0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 -- 172.21.15.186:0/1764971930 shutdown_connections 2024-03-28T15:10:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff20006c530 0x7ff20006e9f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff21c150fd0 0x7ff21c142b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff21c0c2800 0x7ff21c153aa0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 --2- 172.21.15.186:0/1764971930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff21c06c940 0x7ff21c0c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 -- 172.21.15.186:0/1764971930 >> 172.21.15.186:0/1764971930 conn(0x7ff21c06c010 msgr2=0x7ff21c14b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 -- 172.21.15.186:0/1764971930 shutdown_connections 2024-03-28T15:10:49.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.149+0000 7ff2238c0640 1 -- 172.21.15.186:0/1764971930 wait complete. 2024-03-28T15:10:49.162 INFO:tasks.ceph:Scrubbing osd.4 2024-03-28T15:10:49.163 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-28T15:10:49.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.267+0000 7f43159a0640 1 Processor -- start 2024-03-28T15:10:49.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.267+0000 7f43159a0640 1 -- start start 2024-03-28T15:10:49.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f43159a0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f43100678d0 0x7f431006b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f43159a0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4310073a60 0x7f43100c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f43159a0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4310073ef0 con 0x7f43100674f0 2024-03-28T15:10:49.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f43159a0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f4310072d30 con 0x7f4310073a60 2024-03-28T15:10:49.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f43159a0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f43100c8130 con 0x7f43100678d0 2024-03-28T15:10:49.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f431519f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4310073a60 0x7f43100c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f430ffff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f43100678d0 0x7f431006b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f430ffff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f43100678d0 0x7f431006b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.186:56444/0 (socket says 172.21.15.186:56444) 2024-03-28T15:10:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f431519f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4310073a60 0x7f43100c78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49650/0 (socket says 172.21.15.186:49650) 2024-03-28T15:10:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f430ffff640 1 -- 172.21.15.186:0/3546972529 learned_addr learned my addr 172.21.15.186:0/3546972529 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f430ffff640 1 -- 172.21.15.186:0/3546972529 >> v2:172.21.15.186:3300/0 conn(0x7f4310073a60 msgr2=0x7f43100c78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f430ffff640 1 --2- 172.21.15.186:0/3546972529 >> v2:172.21.15.186:3300/0 conn(0x7f4310073a60 0x7f43100c78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f430ffff640 1 -- 172.21.15.186:0/3546972529 >> v1:172.21.15.191:6789/0 conn(0x7f43100674f0 legacy=0x7f431015b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.268+0000 7f430ffff640 1 -- 172.21.15.186:0/3546972529 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43100c8410 con 0x7f43100678d0 2024-03-28T15:10:49.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.269+0000 7f430ffff640 1 --2- 172.21.15.186:0/3546972529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43100678d0 0x7f431006b460 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4310071ec0 tx=0x7f42fc02ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39ed42c7a7c26488 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.269+0000 7f430f7fe640 1 -- 172.21.15.186:0/3546972529 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc02f850 con 0x7f43100678d0 2024-03-28T15:10:49.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.269+0000 7f430f7fe640 1 -- 172.21.15.186:0/3546972529 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42fc02f9f0 con 0x7f43100678d0 2024-03-28T15:10:49.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.269+0000 7f430f7fe640 1 -- 172.21.15.186:0/3546972529 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc02fdc0 con 0x7f43100678d0 2024-03-28T15:10:49.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.269+0000 7f43159a0640 1 -- 172.21.15.186:0/3546972529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43100678d0 msgr2=0x7f431006b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.269+0000 7f43159a0640 1 --2- 172.21.15.186:0/3546972529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43100678d0 0x7f431006b460 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4310071ec0 tx=0x7f42fc02ba30 comp rx=0 tx=0).stop 2024-03-28T15:10:49.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.269+0000 7f43159a0640 1 -- 172.21.15.186:0/3546972529 shutdown_connections 2024-03-28T15:10:49.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.269+0000 7f43159a0640 1 --2- 172.21.15.186:0/3546972529 >> v2:172.21.15.186:3300/0 conn(0x7f4310073a60 0x7f43100c78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.269+0000 7f43159a0640 1 --2- 172.21.15.186:0/3546972529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43100678d0 0x7f431006b460 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.269+0000 7f43159a0640 1 -- 172.21.15.186:0/3546972529 >> 172.21.15.186:0/3546972529 conn(0x7f431006fb90 msgr2=0x7f431006ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.269+0000 7f43159a0640 1 -- 172.21.15.186:0/3546972529 shutdown_connections 2024-03-28T15:10:49.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.270+0000 7f43159a0640 1 -- 172.21.15.186:0/3546972529 wait complete. 2024-03-28T15:10:49.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.271+0000 7f43159a0640 1 Processor -- start 2024-03-28T15:10:49.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.271+0000 7f43159a0640 1 -- start start 2024-03-28T15:10:49.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.271+0000 7f43159a0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43100674f0 0x7f431015def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.271+0000 7f43159a0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43100678d0 0x7f4310149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.271+0000 7f43159a0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4310073a60 0x7f4310149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.271+0000 7f43159a0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4310073540 con 0x7f43100678d0 2024-03-28T15:10:49.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.271+0000 7f43159a0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4310073260 con 0x7f4310073a60 2024-03-28T15:10:49.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.271+0000 7f43159a0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f43100c7e00 con 0x7f43100674f0 2024-03-28T15:10:49.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43100678d0 0x7f4310149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43100678d0 0x7f4310149730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49660/0 (socket says 172.21.15.186:49660) 2024-03-28T15:10:49.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430ffff640 1 -- 172.21.15.186:0/2797611963 learned_addr learned my addr 172.21.15.186:0/2797611963 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:49.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f431499e640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43100674f0 0x7f431015def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f431519f640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4310073a60 0x7f4310149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430ffff640 1 -- 172.21.15.186:0/2797611963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43100674f0 msgr2=0x7f431015def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430ffff640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43100674f0 0x7f431015def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430ffff640 1 -- 172.21.15.186:0/2797611963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4310073a60 msgr2=0x7f4310149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430ffff640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4310073a60 0x7f4310149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430ffff640 1 -- 172.21.15.186:0/2797611963 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431014afc0 con 0x7f43100678d0 2024-03-28T15:10:49.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430ffff640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43100678d0 0x7f4310149730 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f42fc000c00 tx=0x7f42fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430d7fa640 1 -- 172.21.15.186:0/2797611963 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc047070 con 0x7f43100678d0 2024-03-28T15:10:49.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430d7fa640 1 -- 172.21.15.186:0/2797611963 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42fc037d30 con 0x7f43100678d0 2024-03-28T15:10:49.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f43159a0640 1 -- 172.21.15.186:0/2797611963 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431014b2a0 con 0x7f43100678d0 2024-03-28T15:10:49.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f430d7fa640 1 -- 172.21.15.186:0/2797611963 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc0426c0 con 0x7f43100678d0 2024-03-28T15:10:49.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.272+0000 7f43159a0640 1 -- 172.21.15.186:0/2797611963 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43101f1ad0 con 0x7f43100678d0 2024-03-28T15:10:49.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.273+0000 7f43159a0640 1 -- 172.21.15.186:0/2797611963 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f43100686d0 con 0x7f43100678d0 2024-03-28T15:10:49.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.274+0000 7f430d7fa640 1 -- 172.21.15.186:0/2797611963 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f42fc04a070 con 0x7f43100678d0 2024-03-28T15:10:49.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.274+0000 7f430d7fa640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f42f406c530 0x7f42f406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.274+0000 7f430d7fa640 1 -- 172.21.15.186:0/2797611963 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f42fc0adc80 con 0x7f43100678d0 2024-03-28T15:10:49.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.274+0000 7f431499e640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f42f406c530 0x7f42f406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.274+0000 7f430d7fa640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] conn(0x7f42f4074be0 0x7f42f4077060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.274+0000 7f430d7fa640 1 -- 172.21.15.186:0/2797611963 --> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42fc045e70 con 0x7f42f4074be0 2024-03-28T15:10:49.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.274+0000 7f430d7fa640 1 -- 172.21.15.186:0/2797611963 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f42fc0b0380 con 0x7f43100678d0 2024-03-28T15:10:49.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.274+0000 7f431519f640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] conn(0x7f42f4074be0 0x7f42f4077060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.274+0000 7f431499e640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f42f406c530 0x7f42f406e9f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4304001780 tx=0x7f4304009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.275+0000 7f431519f640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] conn(0x7f42f4074be0 0x7f42f4077060 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.275+0000 7f430d7fa640 1 -- 172.21.15.186:0/2797611963 <== osd.4 v2:172.21.15.191:6824/1698221714 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42fc045e70 con 0x7f42f4074be0 2024-03-28T15:10:49.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.288+0000 7f43159a0640 1 -- 172.21.15.186:0/2797611963 --> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f43100c7a60 con 0x7f42f4074be0 2024-03-28T15:10:49.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.289+0000 7f430d7fa640 1 -- 172.21.15.186:0/2797611963 <== osd.4 v2:172.21.15.191:6824/1698221714 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f43100c7a60 con 0x7f42f4074be0 2024-03-28T15:10:49.289 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-28T15:10:49.290 INFO:teuthology.orchestra.run.smithi186.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T15:10:49.290 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-28T15:10:49.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.289+0000 7f43159a0640 1 -- 172.21.15.186:0/2797611963 >> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] conn(0x7f42f4074be0 msgr2=0x7f42f4077060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.289+0000 7f43159a0640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.191:6824/1698221714,v1:172.21.15.191:6825/1698221714] conn(0x7f42f4074be0 0x7f42f4077060 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.289+0000 7f43159a0640 1 -- 172.21.15.186:0/2797611963 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f42f406c530 msgr2=0x7f42f406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.289+0000 7f43159a0640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f42f406c530 0x7f42f406e9f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4304001780 tx=0x7f4304009040 comp rx=0 tx=0).stop 2024-03-28T15:10:49.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.289+0000 7f43159a0640 1 -- 172.21.15.186:0/2797611963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43100678d0 msgr2=0x7f4310149730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.289+0000 7f43159a0640 1 --2- 172.21.15.186:0/2797611963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43100678d0 0x7f4310149730 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f42fc000c00 tx=0x7f42fc004730 comp rx=0 tx=0).stop 2024-03-28T15:10:49.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.289+0000 7f431519f640 1 -- 172.21.15.186:0/2797611963 reap_dead start 2024-03-28T15:10:49.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.289+0000 7f43159a0640 1 -- 172.21.15.186:0/2797611963 shutdown_connections 2024-03-28T15:10:49.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.289+0000 7f43159a0640 1 -- 172.21.15.186:0/2797611963 >> 172.21.15.186:0/2797611963 conn(0x7f431006fb90 msgr2=0x7f43100c2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:49.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.290+0000 7f43159a0640 1 -- 172.21.15.186:0/2797611963 shutdown_connections 2024-03-28T15:10:49.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.290+0000 7f43159a0640 1 -- 172.21.15.186:0/2797611963 wait complete. 2024-03-28T15:10:49.301 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-28T15:10:49.402 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.402+0000 7f71082fa640 1 Processor -- start 2024-03-28T15:10:49.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.402+0000 7f71082fa640 1 -- start start 2024-03-28T15:10:49.404 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f71082fa640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7100072780 0x7f71000d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.404 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f71082fa640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f710015a950 con 0x7f7100154af0 2024-03-28T15:10:49.404 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f71082fa640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f710015abc0 con 0x7f71000cc2b0 2024-03-28T15:10:49.404 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f71082fa640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f710015aea0 con 0x7f7100072780 2024-03-28T15:10:49.404 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f7106096640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f7100154af0 0x7f71000c5da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:60456/0 (socket says 172.21.15.186:60456) 2024-03-28T15:10:49.404 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f7106096640 1 -- 172.21.15.186:0/1000664194 learned_addr learned my addr 172.21.15.186:0/1000664194 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:49.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f7105895640 1 --2- 172.21.15.186:0/1000664194 >> v2:172.21.15.191:3300/0 conn(0x7f7100072780 0x7f71000d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f7105094640 1 -- 172.21.15.186:0/1000664194 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217065572 0 0) 0x7f710015a950 con 0x7f7100154af0 2024-03-28T15:10:49.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f7105094640 1 -- 172.21.15.186:0/1000664194 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70ec003520 con 0x7f7100154af0 2024-03-28T15:10:49.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f7105094640 1 -- 172.21.15.186:0/1000664194 <== mon.0 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3033515986 0 0) 0x7f70ec003520 con 0x7f7100154af0 2024-03-28T15:10:49.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f7105895640 1 -- 172.21.15.186:0/1000664194 >> v1:172.21.15.191:6789/0 conn(0x7f71000cc2b0 legacy=0x7f710015de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f7105895640 1 -- 172.21.15.186:0/1000664194 >> v1:172.21.15.186:6789/0 conn(0x7f7100154af0 legacy=0x7f71000c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.405 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.403+0000 7f7105895640 1 -- 172.21.15.186:0/1000664194 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710015b180 con 0x7f7100072780 2024-03-28T15:10:49.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.404+0000 7f7105895640 1 --2- 172.21.15.186:0/1000664194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100072780 0x7f71000d0610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f70f00097a0 tx=0x7f70f002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a4bb453772b969 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.404+0000 7f7105094640 1 -- 172.21.15.186:0/1000664194 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc001670 con 0x7f7100072780 2024-03-28T15:10:49.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.404+0000 7f7105094640 1 -- 172.21.15.186:0/1000664194 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70f0037070 con 0x7f7100072780 2024-03-28T15:10:49.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.404+0000 7f7105094640 1 -- 172.21.15.186:0/1000664194 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f00317b0 con 0x7f7100072780 2024-03-28T15:10:49.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.405+0000 7f71082fa640 1 -- 172.21.15.186:0/1000664194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100072780 msgr2=0x7f71000d0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.405+0000 7f71082fa640 1 --2- 172.21.15.186:0/1000664194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100072780 0x7f71000d0610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f70f00097a0 tx=0x7f70f002b8b0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.405+0000 7f71082fa640 1 -- 172.21.15.186:0/1000664194 shutdown_connections 2024-03-28T15:10:49.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.405+0000 7f71082fa640 1 --2- 172.21.15.186:0/1000664194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100072780 0x7f71000d0610 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.405+0000 7f71082fa640 1 -- 172.21.15.186:0/1000664194 >> 172.21.15.186:0/1000664194 conn(0x7f7100073580 msgr2=0x7f7100073990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:49.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.405+0000 7f71082fa640 1 -- 172.21.15.186:0/1000664194 shutdown_connections 2024-03-28T15:10:49.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.405+0000 7f71082fa640 1 -- 172.21.15.186:0/1000664194 wait complete. 2024-03-28T15:10:49.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.406+0000 7f71082fa640 1 Processor -- start 2024-03-28T15:10:49.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.406+0000 7f71082fa640 1 -- start start 2024-03-28T15:10:49.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f71082fa640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100072780 0x7f710006b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f71082fa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71000cc2b0 0x7f710015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f71082fa640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7100154af0 0x7f710015bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f71082fa640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71000cd800 con 0x7f71000cc2b0 2024-03-28T15:10:49.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f71082fa640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7100154ed0 con 0x7f7100072780 2024-03-28T15:10:49.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f71082fa640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f710015e540 con 0x7f7100154af0 2024-03-28T15:10:49.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7105895640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71000cc2b0 0x7f710015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7105895640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71000cc2b0 0x7f710015d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49666/0 (socket says 172.21.15.186:49666) 2024-03-28T15:10:49.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7105895640 1 -- 172.21.15.186:0/3616449216 learned_addr learned my addr 172.21.15.186:0/3616449216 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:49.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7106096640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100072780 0x7f710006b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7106897640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7100154af0 0x7f710015bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7105895640 1 -- 172.21.15.186:0/3616449216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7100154af0 msgr2=0x7f710015bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.411 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7105895640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7100154af0 0x7f710015bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7105895640 1 -- 172.21.15.186:0/3616449216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100072780 msgr2=0x7f710006b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7105895640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100072780 0x7f710006b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7105895640 1 -- 172.21.15.186:0/3616449216 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71001497b0 con 0x7f71000cc2b0 2024-03-28T15:10:49.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7106897640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7100154af0 0x7f710015bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:10:49.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f7105895640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71000cc2b0 0x7f710015d9b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f70f0034040 tx=0x7f70f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f70f6ffd640 1 -- 172.21.15.186:0/3616449216 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f004a070 con 0x7f71000cc2b0 2024-03-28T15:10:49.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f70f6ffd640 1 -- 172.21.15.186:0/3616449216 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70f0045420 con 0x7f71000cc2b0 2024-03-28T15:10:49.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f70f6ffd640 1 -- 172.21.15.186:0/3616449216 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f00457f0 con 0x7f71000cc2b0 2024-03-28T15:10:49.412 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.407+0000 7f71082fa640 1 -- 172.21.15.186:0/3616449216 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71001509d0 con 0x7f71000cc2b0 2024-03-28T15:10:49.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.408+0000 7f71082fa640 1 -- 172.21.15.186:0/3616449216 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f710014fb90 con 0x7f71000cc2b0 2024-03-28T15:10:49.413 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.409+0000 7f71082fa640 1 -- 172.21.15.186:0/3616449216 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7100069570 con 0x7f71000cc2b0 2024-03-28T15:10:49.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.409+0000 7f70f6ffd640 1 -- 172.21.15.186:0/3616449216 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f70f004d070 con 0x7f71000cc2b0 2024-03-28T15:10:49.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.409+0000 7f70f6ffd640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f70e406c530 0x7f70e406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.410+0000 7f7106096640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f70e406c530 0x7f70e406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.410+0000 7f70f6ffd640 1 -- 172.21.15.186:0/3616449216 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f70f00ad1a0 con 0x7f71000cc2b0 2024-03-28T15:10:49.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.410+0000 7f7106096640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f70e406c530 0x7f70e406e9f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f70fc005880 tx=0x7f70fc0057f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.414+0000 7f70f6ffd640 1 -- 172.21.15.186:0/3616449216 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f70f0040070 con 0x7f71000cc2b0 2024-03-28T15:10:49.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.593+0000 7f71082fa640 1 -- 172.21.15.186:0/3616449216 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f71000ca4f0 con 0x7f70e406c530 2024-03-28T15:10:49.594 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.594+0000 7f70f6ffd640 1 -- 172.21.15.186:0/3616449216 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f71000ca4f0 con 0x7f70e406c530 2024-03-28T15:10:49.594 INFO:teuthology.orchestra.run.smithi186.stderr:instructed osd(s) 4 to deep-scrub 2024-03-28T15:10:49.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 -- 172.21.15.186:0/3616449216 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f70e406c530 msgr2=0x7f70e406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.596 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f70e406c530 0x7f70e406e9f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f70fc005880 tx=0x7f70fc0057f0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 -- 172.21.15.186:0/3616449216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71000cc2b0 msgr2=0x7f710015d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71000cc2b0 0x7f710015d9b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f70f0034040 tx=0x7f70f00046c0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 -- 172.21.15.186:0/3616449216 shutdown_connections 2024-03-28T15:10:49.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f70e406c530 0x7f70e406e9f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7100154af0 0x7f710015bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71000cc2b0 0x7f710015d9b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.597 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 --2- 172.21.15.186:0/3616449216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100072780 0x7f710006b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.598 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 -- 172.21.15.186:0/3616449216 >> 172.21.15.186:0/3616449216 conn(0x7f7100073580 msgr2=0x7f71000b4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:49.598 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 -- 172.21.15.186:0/3616449216 shutdown_connections 2024-03-28T15:10:49.598 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.596+0000 7f71082fa640 1 -- 172.21.15.186:0/3616449216 wait complete. 2024-03-28T15:10:49.608 INFO:tasks.ceph:Scrubbing osd.5 2024-03-28T15:10:49.608 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-28T15:10:49.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.713+0000 7fadb7e4e640 1 Processor -- start 2024-03-28T15:10:49.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.713+0000 7fadb7e4e640 1 -- start start 2024-03-28T15:10:49.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.714+0000 7fadb7e4e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fadb00cc180 0x7fadb00cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.714+0000 7fadb7e4e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadb00cac20 con 0x7fadb0154b10 2024-03-28T15:10:49.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.714+0000 7fadb7e4e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadb00caf00 con 0x7fadb015e850 2024-03-28T15:10:49.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.714+0000 7fadb7e4e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fadb00cc660 con 0x7fadb00cc180 2024-03-28T15:10:49.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.714+0000 7fadb63eb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fadb00cc180 0x7fadb00cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.714+0000 7fadb5bea640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fadb0154b10 0x7fadb00d06c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:60466/0 (socket says 172.21.15.186:60466) 2024-03-28T15:10:49.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.714+0000 7fadb5bea640 1 -- 172.21.15.186:0/2390523396 learned_addr learned my addr 172.21.15.186:0/2390523396 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:49.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.714+0000 7fadb63eb640 1 -- 172.21.15.186:0/2390523396 >> v1:172.21.15.191:6789/0 conn(0x7fadb015e850 legacy=0x7fadb00c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.714+0000 7fadb4be8640 1 -- 172.21.15.186:0/2390523396 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 63829790 0 0) 0x7fadb00caf00 con 0x7fadb015e850 2024-03-28T15:10:49.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.714+0000 7fadb63eb640 1 -- 172.21.15.186:0/2390523396 >> v1:172.21.15.186:6789/0 conn(0x7fadb0154b10 legacy=0x7fadb00d06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.714+0000 7fadb63eb640 1 -- 172.21.15.186:0/2390523396 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb00c5360 con 0x7fadb00cc180 2024-03-28T15:10:49.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.715+0000 7fadb63eb640 1 --2- 172.21.15.186:0/2390523396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadb00cc180 0x7fadb00cb9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fada8009d90 tx=0x7fada802bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa9baaa2f97af63 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.715+0000 7fadb4be8640 1 -- 172.21.15.186:0/2390523396 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8039070 con 0x7fadb00cc180 2024-03-28T15:10:49.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.715+0000 7fadb4be8640 1 -- 172.21.15.186:0/2390523396 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada802ca20 con 0x7fadb00cc180 2024-03-28T15:10:49.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.715+0000 7fadb4be8640 1 -- 172.21.15.186:0/2390523396 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada802cdf0 con 0x7fadb00cc180 2024-03-28T15:10:49.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.716+0000 7fadb7e4e640 1 -- 172.21.15.186:0/2390523396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadb00cc180 msgr2=0x7fadb00cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.716+0000 7fadb7e4e640 1 --2- 172.21.15.186:0/2390523396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadb00cc180 0x7fadb00cb9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fada8009d90 tx=0x7fada802bcf0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.716+0000 7fadb7e4e640 1 -- 172.21.15.186:0/2390523396 shutdown_connections 2024-03-28T15:10:49.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.716+0000 7fadb7e4e640 1 --2- 172.21.15.186:0/2390523396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadb00cc180 0x7fadb00cb9c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.716+0000 7fadb7e4e640 1 -- 172.21.15.186:0/2390523396 >> 172.21.15.186:0/2390523396 conn(0x7fadb00733a0 msgr2=0x7fadb00737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:49.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.716+0000 7fadb7e4e640 1 -- 172.21.15.186:0/2390523396 shutdown_connections 2024-03-28T15:10:49.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.716+0000 7fadb7e4e640 1 -- 172.21.15.186:0/2390523396 wait complete. 2024-03-28T15:10:49.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.717+0000 7fadb7e4e640 1 Processor -- start 2024-03-28T15:10:49.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.717+0000 7fadb7e4e640 1 -- start start 2024-03-28T15:10:49.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.717+0000 7fadb7e4e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb00cc180 0x7fadb014e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.717+0000 7fadb7e4e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fadb0154b10 0x7fadb00ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.717+0000 7fadb7e4e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadb015e850 0x7fadb01f06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.717+0000 7fadb7e4e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fadb015edb0 con 0x7fadb015e850 2024-03-28T15:10:49.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.717+0000 7fadb7e4e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fadb00cbf00 con 0x7fadb00cc180 2024-03-28T15:10:49.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.717+0000 7fadb7e4e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fadb00c5110 con 0x7fadb0154b10 2024-03-28T15:10:49.722 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb63eb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadb015e850 0x7fadb01f06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb63eb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadb015e850 0x7fadb01f06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49678/0 (socket says 172.21.15.186:49678) 2024-03-28T15:10:49.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb63eb640 1 -- 172.21.15.186:0/3875139460 learned_addr learned my addr 172.21.15.186:0/3875139460 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:49.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb5bea640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb00cc180 0x7fadb014e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb53e9640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fadb0154b10 0x7fadb00ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb63eb640 1 -- 172.21.15.186:0/3875139460 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fadb0154b10 msgr2=0x7fadb00ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb63eb640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fadb0154b10 0x7fadb00ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb63eb640 1 -- 172.21.15.186:0/3875139460 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb00cc180 msgr2=0x7fadb014e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb63eb640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb00cc180 0x7fadb014e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb63eb640 1 -- 172.21.15.186:0/3875139460 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb00b5660 con 0x7fadb015e850 2024-03-28T15:10:49.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb63eb640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadb015e850 0x7fadb01f06c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fada8037040 tx=0x7fada8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fada67fc640 1 -- 172.21.15.186:0/3875139460 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8044020 con 0x7fadb015e850 2024-03-28T15:10:49.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fada67fc640 1 -- 172.21.15.186:0/3875139460 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada8034c10 con 0x7fadb015e850 2024-03-28T15:10:49.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb7e4e640 1 -- 172.21.15.186:0/3875139460 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadb00b4ae0 con 0x7fadb015e850 2024-03-28T15:10:49.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fada67fc640 1 -- 172.21.15.186:0/3875139460 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8039040 con 0x7fadb015e850 2024-03-28T15:10:49.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.718+0000 7fadb7e4e640 1 -- 172.21.15.186:0/3875139460 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb00b50a0 con 0x7fadb015e850 2024-03-28T15:10:49.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.719+0000 7fadb7e4e640 1 -- 172.21.15.186:0/3875139460 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fadb015cb20 con 0x7fadb015e850 2024-03-28T15:10:49.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.720+0000 7fada67fc640 1 -- 172.21.15.186:0/3875139460 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fada8005d10 con 0x7fadb015e850 2024-03-28T15:10:49.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.720+0000 7fada67fc640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fad9406c530 0x7fad9406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.720+0000 7fada67fc640 1 -- 172.21.15.186:0/3875139460 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fada80adf80 con 0x7fadb015e850 2024-03-28T15:10:49.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.720+0000 7fadb5bea640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fad9406c530 0x7fad9406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.720+0000 7fada67fc640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] conn(0x7fad94074be0 0x7fad94077060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.720+0000 7fada67fc640 1 -- 172.21.15.186:0/3875139460 --> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fada80020b0 con 0x7fad94074be0 2024-03-28T15:10:49.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.720+0000 7fada67fc640 1 -- 172.21.15.186:0/3875139460 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fada80b0620 con 0x7fadb015e850 2024-03-28T15:10:49.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.721+0000 7fadb5bea640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fad9406c530 0x7fad9406e9f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fadac000f60 tx=0x7fadac007e20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.721+0000 7fadb53e9640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] conn(0x7fad94074be0 0x7fad94077060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.721+0000 7fadb53e9640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] conn(0x7fad94074be0 0x7fad94077060 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.722+0000 7fada67fc640 1 -- 172.21.15.186:0/3875139460 <== osd.5 v2:172.21.15.191:6808/138503776 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fada80020b0 con 0x7fad94074be0 2024-03-28T15:10:49.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.735+0000 7fadb7e4e640 1 -- 172.21.15.186:0/3875139460 --> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fadb0069570 con 0x7fad94074be0 2024-03-28T15:10:49.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.735+0000 7fada67fc640 1 -- 172.21.15.186:0/3875139460 <== osd.5 v2:172.21.15.191:6808/138503776 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fadb0069570 con 0x7fad94074be0 2024-03-28T15:10:49.736 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-28T15:10:49.736 INFO:teuthology.orchestra.run.smithi186.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T15:10:49.736 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-28T15:10:49.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.735+0000 7fadb7e4e640 1 -- 172.21.15.186:0/3875139460 >> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] conn(0x7fad94074be0 msgr2=0x7fad94077060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.735+0000 7fadb7e4e640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.191:6808/138503776,v1:172.21.15.191:6809/138503776] conn(0x7fad94074be0 0x7fad94077060 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.735+0000 7fadb7e4e640 1 -- 172.21.15.186:0/3875139460 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fad9406c530 msgr2=0x7fad9406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.735+0000 7fadb7e4e640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fad9406c530 0x7fad9406e9f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fadac000f60 tx=0x7fadac007e20 comp rx=0 tx=0).stop 2024-03-28T15:10:49.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.735+0000 7fadb7e4e640 1 -- 172.21.15.186:0/3875139460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadb015e850 msgr2=0x7fadb01f06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.735+0000 7fadb7e4e640 1 --2- 172.21.15.186:0/3875139460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadb015e850 0x7fadb01f06c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fada8037040 tx=0x7fada8004a40 comp rx=0 tx=0).stop 2024-03-28T15:10:49.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.735+0000 7fadb63eb640 1 -- 172.21.15.186:0/3875139460 reap_dead start 2024-03-28T15:10:49.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.735+0000 7fadb7e4e640 1 -- 172.21.15.186:0/3875139460 shutdown_connections 2024-03-28T15:10:49.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.735+0000 7fadb7e4e640 1 -- 172.21.15.186:0/3875139460 >> 172.21.15.186:0/3875139460 conn(0x7fadb00733a0 msgr2=0x7fadb00d0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:49.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.736+0000 7fadb7e4e640 1 -- 172.21.15.186:0/3875139460 shutdown_connections 2024-03-28T15:10:49.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.736+0000 7fadb7e4e640 1 -- 172.21.15.186:0/3875139460 wait complete. 2024-03-28T15:10:49.748 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-28T15:10:49.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.850+0000 7f0ee2d7a640 1 Processor -- start 2024-03-28T15:10:49.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.850+0000 7f0ee2d7a640 1 -- start start 2024-03-28T15:10:49.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee2d7a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0edc06e080 0x7f0edc15b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee2d7a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0edc0b06c0 0x7f0edc15dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee2d7a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0edc15e590 con 0x7f0edc06ebe0 2024-03-28T15:10:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee2d7a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f0edc15e830 con 0x7f0edc0b06c0 2024-03-28T15:10:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee2d7a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f0edc15eb10 con 0x7f0edc06e080 2024-03-28T15:10:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee1317640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0edc0b06c0 0x7f0edc15dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee1317640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0edc0b06c0 0x7f0edc15dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49690/0 (socket says 172.21.15.186:49690) 2024-03-28T15:10:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee1317640 1 -- 172.21.15.186:0/2331274582 learned_addr learned my addr 172.21.15.186:0/2331274582 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0edbfff640 1 --2- 172.21.15.186:0/2331274582 >> v2:172.21.15.191:3300/0 conn(0x7f0edc06e080 0x7f0edc15b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee1317640 1 -- 172.21.15.186:0/2331274582 >> v2:172.21.15.191:3300/0 conn(0x7f0edc06e080 msgr2=0x7f0edc15b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:49.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee1317640 1 --2- 172.21.15.186:0/2331274582 >> v2:172.21.15.191:3300/0 conn(0x7f0edc06e080 0x7f0edc15b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee1317640 1 -- 172.21.15.186:0/2331274582 >> v1:172.21.15.191:6789/0 conn(0x7f0edc06ebe0 legacy=0x7f0edc0affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0ee1317640 1 -- 172.21.15.186:0/2331274582 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0edc15edf0 con 0x7f0edc0b06c0 2024-03-28T15:10:49.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.851+0000 7f0edbfff640 1 --2- 172.21.15.186:0/2331274582 >> v2:172.21.15.191:3300/0 conn(0x7f0edc06e080 0x7f0edc15b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:49.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.852+0000 7f0ee1317640 1 --2- 172.21.15.186:0/2331274582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0edc0b06c0 0x7f0edc15dfa0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0edc0ccfe0 tx=0x7f0ed402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c384f7578f28440 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.852+0000 7f0edb7fe640 1 -- 172.21.15.186:0/2331274582 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed402cb20 con 0x7f0edc0b06c0 2024-03-28T15:10:49.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.852+0000 7f0edb7fe640 1 -- 172.21.15.186:0/2331274582 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ed402ccc0 con 0x7f0edc0b06c0 2024-03-28T15:10:49.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.852+0000 7f0edb7fe640 1 -- 172.21.15.186:0/2331274582 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4034610 con 0x7f0edc0b06c0 2024-03-28T15:10:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.853+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2331274582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0edc0b06c0 msgr2=0x7f0edc15dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.853+0000 7f0ee2d7a640 1 --2- 172.21.15.186:0/2331274582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0edc0b06c0 0x7f0edc15dfa0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0edc0ccfe0 tx=0x7f0ed402bf80 comp rx=0 tx=0).stop 2024-03-28T15:10:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.853+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2331274582 shutdown_connections 2024-03-28T15:10:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.853+0000 7f0ee2d7a640 1 --2- 172.21.15.186:0/2331274582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0edc0b06c0 0x7f0edc15dfa0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.853+0000 7f0ee2d7a640 1 --2- 172.21.15.186:0/2331274582 >> v2:172.21.15.191:3300/0 conn(0x7f0edc06e080 0x7f0edc15b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.853+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2331274582 >> 172.21.15.186:0/2331274582 conn(0x7f0edc06c010 msgr2=0x7f0edc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:49.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.853+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2331274582 shutdown_connections 2024-03-28T15:10:49.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.853+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2331274582 wait complete. 2024-03-28T15:10:49.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.854+0000 7f0ee2d7a640 1 Processor -- start 2024-03-28T15:10:49.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.854+0000 7f0ee2d7a640 1 -- start start 2024-03-28T15:10:49.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ee2d7a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0edc06e080 0x7f0edc072590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ee2d7a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0edc06ebe0 0x7f0edc1f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ee2d7a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0edc0b06c0 0x7f0edc1f0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ee2d7a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc15b100 con 0x7f0edc06ebe0 2024-03-28T15:10:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ee2d7a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc15ae20 con 0x7f0edc0b06c0 2024-03-28T15:10:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ee2d7a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f0edc0c86b0 con 0x7f0edc06e080 2024-03-28T15:10:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0edbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0edc06ebe0 0x7f0edc1f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0edbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0edc06ebe0 0x7f0edc1f0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49694/0 (socket says 172.21.15.186:49694) 2024-03-28T15:10:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0edbfff640 1 -- 172.21.15.186:0/2364697736 learned_addr learned my addr 172.21.15.186:0/2364697736 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ee1317640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0edc0b06c0 0x7f0edc1f0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ee0b16640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0edc06e080 0x7f0edc072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0edbfff640 1 -- 172.21.15.186:0/2364697736 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0edc06e080 msgr2=0x7f0edc072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0edbfff640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0edc06e080 0x7f0edc072590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0edbfff640 1 -- 172.21.15.186:0/2364697736 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0edc0b06c0 msgr2=0x7f0edc1f0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:49.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0edbfff640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0edc0b06c0 0x7f0edc1f0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:49.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0edbfff640 1 -- 172.21.15.186:0/2364697736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0edc15bf70 con 0x7f0edc06ebe0 2024-03-28T15:10:49.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0edbfff640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0edc06ebe0 0x7f0edc1f0540 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800e990 tx=0x7f0ec800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ed97fa640 1 -- 172.21.15.186:0/2364697736 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8017070 con 0x7f0edc06ebe0 2024-03-28T15:10:49.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ed97fa640 1 -- 172.21.15.186:0/2364697736 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ec800c9e0 con 0x7f0edc06ebe0 2024-03-28T15:10:49.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2364697736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0edc150ab0 con 0x7f0edc06ebe0 2024-03-28T15:10:49.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ed97fa640 1 -- 172.21.15.186:0/2364697736 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec800cdb0 con 0x7f0edc06ebe0 2024-03-28T15:10:49.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.855+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2364697736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0edc15c510 con 0x7f0edc06ebe0 2024-03-28T15:10:49.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.856+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2364697736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0edc069570 con 0x7f0edc06ebe0 2024-03-28T15:10:49.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.857+0000 7f0ed97fa640 1 -- 172.21.15.186:0/2364697736 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0ec801a070 con 0x7f0edc06ebe0 2024-03-28T15:10:49.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.857+0000 7f0ed97fa640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ec006c530 0x7f0ec006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:49.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.857+0000 7f0ed97fa640 1 -- 172.21.15.186:0/2364697736 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f0ec808b7b0 con 0x7f0edc06ebe0 2024-03-28T15:10:49.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.857+0000 7f0ee0b16640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ec006c530 0x7f0ec006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:49.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.858+0000 7f0ee0b16640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ec006c530 0x7f0ec006e9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0001780 tx=0x7f0ed0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:49.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:49.861+0000 7f0ed97fa640 1 -- 172.21.15.186:0/2364697736 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f0ec80594d0 con 0x7f0edc06ebe0 2024-03-28T15:10:50.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.083+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2364697736 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f0edc073220 con 0x7f0ec006c530 2024-03-28T15:10:50.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.083+0000 7f0ed97fa640 1 -- 172.21.15.186:0/2364697736 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0ed0001420 con 0x7f0ec006c530 2024-03-28T15:10:50.084 INFO:teuthology.orchestra.run.smithi186.stderr:instructed osd(s) 5 to deep-scrub 2024-03-28T15:10:50.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.085+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2364697736 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ec006c530 msgr2=0x7f0ec006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.085+0000 7f0ee2d7a640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ec006c530 0x7f0ec006e9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0001780 tx=0x7f0ed0009040 comp rx=0 tx=0).stop 2024-03-28T15:10:50.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.085+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2364697736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0edc06ebe0 msgr2=0x7f0edc1f0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.085+0000 7f0ee2d7a640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0edc06ebe0 0x7f0edc1f0540 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800e990 tx=0x7f0ec800ee50 comp rx=0 tx=0).stop 2024-03-28T15:10:50.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.085+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2364697736 shutdown_connections 2024-03-28T15:10:50.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.085+0000 7f0ee2d7a640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0ec006c530 0x7f0ec006e9f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.085+0000 7f0ee2d7a640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0edc0b06c0 0x7f0edc1f0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.085+0000 7f0ee2d7a640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0edc06ebe0 0x7f0edc1f0540 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.085+0000 7f0ee2d7a640 1 --2- 172.21.15.186:0/2364697736 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0edc06e080 0x7f0edc072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.085+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2364697736 >> 172.21.15.186:0/2364697736 conn(0x7f0edc06c010 msgr2=0x7f0edc153e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:50.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.086+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2364697736 shutdown_connections 2024-03-28T15:10:50.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.086+0000 7f0ee2d7a640 1 -- 172.21.15.186:0/2364697736 wait complete. 2024-03-28T15:10:50.097 INFO:tasks.ceph:Scrubbing osd.6 2024-03-28T15:10:50.097 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-28T15:10:50.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.203+0000 7f707d00f640 1 Processor -- start 2024-03-28T15:10:50.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.203+0000 7f707d00f640 1 -- start start 2024-03-28T15:10:50.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f707d00f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f70781515b0 0x7f7078157b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f707d00f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780c12a0 0x7f7078158080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f707d00f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f70780b1d30 0x7f70780712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f707d00f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f7078151990 con 0x7f70781515b0 2024-03-28T15:10:50.206 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f707d00f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f707806d130 con 0x7f70780b1d30 2024-03-28T15:10:50.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f707d00f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f707806c920 con 0x7f70780c12a0 2024-03-28T15:10:50.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f7076d9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f70781515b0 0x7f7078157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f7076d9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f70781515b0 0x7f7078157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49704/0 (socket says 172.21.15.186:49704) 2024-03-28T15:10:50.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f7076d9d640 1 -- 172.21.15.186:0/1330529428 learned_addr learned my addr 172.21.15.186:0/1330529428 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:50.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f706e59c640 1 --2- 172.21.15.186:0/1330529428 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780c12a0 0x7f7078158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f707759e640 1 --2- 172.21.15.186:0/1330529428 >> v2:172.21.15.191:3300/0 conn(0x7f70780b1d30 0x7f70780712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f7076d9d640 1 -- 172.21.15.186:0/1330529428 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780c12a0 msgr2=0x7f7078158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:50.207 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f7076d9d640 1 --2- 172.21.15.186:0/1330529428 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780c12a0 0x7f7078158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f7076d9d640 1 -- 172.21.15.186:0/1330529428 >> v2:172.21.15.191:3300/0 conn(0x7f70780b1d30 msgr2=0x7f70780712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:50.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f7076d9d640 1 --2- 172.21.15.186:0/1330529428 >> v2:172.21.15.191:3300/0 conn(0x7f70780b1d30 0x7f70780712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.204+0000 7f7076d9d640 1 -- 172.21.15.186:0/1330529428 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70780c17e0 con 0x7f70781515b0 2024-03-28T15:10:50.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.205+0000 7f7076d9d640 1 --2- 172.21.15.186:0/1330529428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70781515b0 0x7f7078157b40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f707806fcd0 tx=0x7f706802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96acbcd3f2ce3e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.205+0000 7f707659c640 1 -- 172.21.15.186:0/1330529428 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068034070 con 0x7f70781515b0 2024-03-28T15:10:50.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.205+0000 7f707659c640 1 -- 172.21.15.186:0/1330529428 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7068030860 con 0x7f70781515b0 2024-03-28T15:10:50.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.205+0000 7f707659c640 1 -- 172.21.15.186:0/1330529428 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068030c30 con 0x7f70781515b0 2024-03-28T15:10:50.208 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.205+0000 7f707d00f640 1 -- 172.21.15.186:0/1330529428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70781515b0 msgr2=0x7f7078157b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.205+0000 7f707d00f640 1 --2- 172.21.15.186:0/1330529428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70781515b0 0x7f7078157b40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f707806fcd0 tx=0x7f706802b9a0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.205+0000 7f707d00f640 1 -- 172.21.15.186:0/1330529428 shutdown_connections 2024-03-28T15:10:50.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.205+0000 7f707d00f640 1 --2- 172.21.15.186:0/1330529428 >> v2:172.21.15.191:3300/0 conn(0x7f70780b1d30 0x7f70780712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.205+0000 7f707d00f640 1 --2- 172.21.15.186:0/1330529428 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780c12a0 0x7f7078158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.205+0000 7f707d00f640 1 --2- 172.21.15.186:0/1330529428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70781515b0 0x7f7078157b40 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.205+0000 7f707d00f640 1 -- 172.21.15.186:0/1330529428 >> 172.21.15.186:0/1330529428 conn(0x7f70780c8a20 msgr2=0x7f70780c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:50.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.206+0000 7f707d00f640 1 -- 172.21.15.186:0/1330529428 shutdown_connections 2024-03-28T15:10:50.209 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.206+0000 7f707d00f640 1 -- 172.21.15.186:0/1330529428 wait complete. 2024-03-28T15:10:50.212 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.207+0000 7f707d00f640 1 Processor -- start 2024-03-28T15:10:50.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.207+0000 7f707d00f640 1 -- start start 2024-03-28T15:10:50.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f707d00f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780b1d30 0x7f707806d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f707d00f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70780c12a0 0x7f70781f1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f707d00f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70781515b0 0x7f7078145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f707d00f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7078072d20 con 0x7f70780c12a0 2024-03-28T15:10:50.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f707d00f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f707806ce50 con 0x7f70781515b0 2024-03-28T15:10:50.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f707d00f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f7078073d70 con 0x7f70780b1d30 2024-03-28T15:10:50.213 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70780c12a0 0x7f70781f1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70780c12a0 0x7f70781f1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49706/0 (socket says 172.21.15.186:49706) 2024-03-28T15:10:50.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706e59c640 1 -- 172.21.15.186:0/2815445716 learned_addr learned my addr 172.21.15.186:0/2815445716 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:50.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f707759e640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70781515b0 0x7f7078145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f7076d9d640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780b1d30 0x7f707806d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706e59c640 1 -- 172.21.15.186:0/2815445716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780b1d30 msgr2=0x7f707806d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706e59c640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780b1d30 0x7f707806d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706e59c640 1 -- 172.21.15.186:0/2815445716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70781515b0 msgr2=0x7f7078145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.214 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706e59c640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70781515b0 0x7f7078145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706e59c640 1 -- 172.21.15.186:0/2815445716 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707806d8d0 con 0x7f70780c12a0 2024-03-28T15:10:50.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706e59c640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70780c12a0 0x7f70781f1170 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f706000e9e0 tx=0x7f706000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706ffff640 1 -- 172.21.15.186:0/2815445716 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060017070 con 0x7f70780c12a0 2024-03-28T15:10:50.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706ffff640 1 -- 172.21.15.186:0/2815445716 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f706000c9d0 con 0x7f70780c12a0 2024-03-28T15:10:50.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f707d00f640 1 -- 172.21.15.186:0/2815445716 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70781f06b0 con 0x7f70780c12a0 2024-03-28T15:10:50.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.208+0000 7f706ffff640 1 -- 172.21.15.186:0/2815445716 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706000cdc0 con 0x7f70780c12a0 2024-03-28T15:10:50.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.209+0000 7f707d00f640 1 -- 172.21.15.186:0/2815445716 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f707806db70 con 0x7f70780c12a0 2024-03-28T15:10:50.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.209+0000 7f707d00f640 1 -- 172.21.15.186:0/2815445716 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7078157090 con 0x7f70780c12a0 2024-03-28T15:10:50.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.210+0000 7f706ffff640 1 -- 172.21.15.186:0/2815445716 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f706001a070 con 0x7f70780c12a0 2024-03-28T15:10:50.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.210+0000 7f706ffff640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f705c06c530 0x7f705c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.210+0000 7f7076d9d640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f705c06c530 0x7f705c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.210+0000 7f706ffff640 1 -- 172.21.15.186:0/2815445716 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f706008b850 con 0x7f70780c12a0 2024-03-28T15:10:50.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.211+0000 7f706ffff640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] conn(0x7f705c074bd0 0x7f705c077050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.211+0000 7f706ffff640 1 -- 172.21.15.186:0/2815445716 --> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f705c077720 con 0x7f705c074bd0 2024-03-28T15:10:50.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.211+0000 7f706ffff640 1 -- 172.21.15.186:0/2815445716 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f70600902c0 con 0x7f70780c12a0 2024-03-28T15:10:50.216 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.211+0000 7f707759e640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] conn(0x7f705c074bd0 0x7f705c077050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.211+0000 7f7076d9d640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f705c06c530 0x7f705c06e9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7068036000 tx=0x7f7068008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.211+0000 7f707759e640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] conn(0x7f705c074bd0 0x7f705c077050 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.212+0000 7f706ffff640 1 -- 172.21.15.186:0/2815445716 <== osd.6 v2:172.21.15.191:6800/1108197747 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f705c077720 con 0x7f705c074bd0 2024-03-28T15:10:50.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.225+0000 7f707d00f640 1 -- 172.21.15.186:0/2815445716 --> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f70781f16b0 con 0x7f705c074bd0 2024-03-28T15:10:50.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.225+0000 7f706ffff640 1 -- 172.21.15.186:0/2815445716 <== osd.6 v2:172.21.15.191:6800/1108197747 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7070015370 con 0x7f705c074bd0 2024-03-28T15:10:50.226 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-28T15:10:50.226 INFO:teuthology.orchestra.run.smithi186.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T15:10:50.226 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-28T15:10:50.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.226+0000 7f707d00f640 1 -- 172.21.15.186:0/2815445716 >> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] conn(0x7f705c074bd0 msgr2=0x7f705c077050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.226+0000 7f707d00f640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.191:6800/1108197747,v1:172.21.15.191:6801/1108197747] conn(0x7f705c074bd0 0x7f705c077050 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.226+0000 7f707d00f640 1 -- 172.21.15.186:0/2815445716 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f705c06c530 msgr2=0x7f705c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.226+0000 7f707d00f640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f705c06c530 0x7f705c06e9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7068036000 tx=0x7f7068008030 comp rx=0 tx=0).stop 2024-03-28T15:10:50.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.226+0000 7f707d00f640 1 -- 172.21.15.186:0/2815445716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70780c12a0 msgr2=0x7f70781f1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.226+0000 7f707d00f640 1 --2- 172.21.15.186:0/2815445716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70780c12a0 0x7f70781f1170 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f706000e9e0 tx=0x7f706000eea0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.226+0000 7f707759e640 1 -- 172.21.15.186:0/2815445716 reap_dead start 2024-03-28T15:10:50.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.226+0000 7f707d00f640 1 -- 172.21.15.186:0/2815445716 shutdown_connections 2024-03-28T15:10:50.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.226+0000 7f707d00f640 1 -- 172.21.15.186:0/2815445716 >> 172.21.15.186:0/2815445716 conn(0x7f70780c8a20 msgr2=0x7f707814f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:50.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.226+0000 7f707d00f640 1 -- 172.21.15.186:0/2815445716 shutdown_connections 2024-03-28T15:10:50.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.226+0000 7f707d00f640 1 -- 172.21.15.186:0/2815445716 wait complete. 2024-03-28T15:10:50.237 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-28T15:10:50.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.338+0000 7fccc0681640 1 Processor -- start 2024-03-28T15:10:50.339 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.338+0000 7fccc0681640 1 -- start start 2024-03-28T15:10:50.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.339+0000 7fccc0681640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fccb8154b60 0x7fccb815a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.340 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.339+0000 7fccc0681640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fccb80cb3f0 0x7fccb8071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.339+0000 7fccc0681640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccb80cd620 con 0x7fccb80cafd0 2024-03-28T15:10:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.339+0000 7fccc0681640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fccb8072820 con 0x7fccb80cb3f0 2024-03-28T15:10:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.339+0000 7fccc0681640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fccb815b760 con 0x7fccb8154b60 2024-03-28T15:10:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.339+0000 7fccbec1e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fccb80cb3f0 0x7fccb8071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.339+0000 7fccbdc1c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fccb80cafd0 0x7fccb80ccf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:60470/0 (socket says 172.21.15.186:60470) 2024-03-28T15:10:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.339+0000 7fccbdc1c640 1 -- 172.21.15.186:0/3991752199 learned_addr learned my addr 172.21.15.186:0/3991752199 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.339+0000 7fccbec1e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fccb80cb3f0 0x7fccb8071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49712/0 (socket says 172.21.15.186:49712) 2024-03-28T15:10:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.339+0000 7fccbe41d640 1 --2- 172.21.15.186:0/3991752199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fccb8154b60 0x7fccb815a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.340+0000 7fccbec1e640 1 -- 172.21.15.186:0/3991752199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fccb8154b60 msgr2=0x7fccb815a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.340+0000 7fccbec1e640 1 --2- 172.21.15.186:0/3991752199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fccb8154b60 0x7fccb815a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.340+0000 7fccbec1e640 1 -- 172.21.15.186:0/3991752199 >> v1:172.21.15.186:6789/0 conn(0x7fccb80cafd0 legacy=0x7fccb80ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.340+0000 7fccbd41b640 1 -- 172.21.15.186:0/3991752199 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4132637875 0 0) 0x7fcca8001110 con 0x7fccb80cafd0 2024-03-28T15:10:50.342 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.340+0000 7fccbec1e640 1 -- 172.21.15.186:0/3991752199 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb815aec0 con 0x7fccb80cb3f0 2024-03-28T15:10:50.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.340+0000 7fccbec1e640 1 --2- 172.21.15.186:0/3991752199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb80cb3f0 0x7fccb8071a80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fccb00097a0 tx=0x7fccb002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60386404aba45fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.341+0000 7fccbd41b640 1 -- 172.21.15.186:0/3991752199 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0034070 con 0x7fccb80cb3f0 2024-03-28T15:10:50.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.341+0000 7fccbd41b640 1 -- 172.21.15.186:0/3991752199 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb0030610 con 0x7fccb80cb3f0 2024-03-28T15:10:50.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.341+0000 7fccbd41b640 1 -- 172.21.15.186:0/3991752199 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb00309e0 con 0x7fccb80cb3f0 2024-03-28T15:10:50.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.341+0000 7fccc0681640 1 -- 172.21.15.186:0/3991752199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb80cb3f0 msgr2=0x7fccb8071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.341+0000 7fccc0681640 1 --2- 172.21.15.186:0/3991752199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb80cb3f0 0x7fccb8071a80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fccb00097a0 tx=0x7fccb002b960 comp rx=0 tx=0).stop 2024-03-28T15:10:50.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.341+0000 7fccc0681640 1 -- 172.21.15.186:0/3991752199 shutdown_connections 2024-03-28T15:10:50.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.341+0000 7fccc0681640 1 --2- 172.21.15.186:0/3991752199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb80cb3f0 0x7fccb8071a80 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.341+0000 7fccc0681640 1 --2- 172.21.15.186:0/3991752199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fccb8154b60 0x7fccb815a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.341+0000 7fccc0681640 1 -- 172.21.15.186:0/3991752199 >> 172.21.15.186:0/3991752199 conn(0x7fccb8073b80 msgr2=0x7fccb806b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:50.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.341+0000 7fccc0681640 1 -- 172.21.15.186:0/3991752199 shutdown_connections 2024-03-28T15:10:50.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.341+0000 7fccc0681640 1 -- 172.21.15.186:0/3991752199 wait complete. 2024-03-28T15:10:50.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.342+0000 7fccc0681640 1 Processor -- start 2024-03-28T15:10:50.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.342+0000 7fccc0681640 1 -- start start 2024-03-28T15:10:50.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.343+0000 7fccc0681640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fccb80cafd0 0x7fccb8149960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.343+0000 7fccc0681640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fccb80cb3f0 0x7fccb80b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.343+0000 7fccc0681640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb8154b60 0x7fccb80c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.343+0000 7fccc0681640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fccb80cc710 con 0x7fccb8154b60 2024-03-28T15:10:50.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.343+0000 7fccc0681640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fccb815b480 con 0x7fccb80cafd0 2024-03-28T15:10:50.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.343+0000 7fccc0681640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fccb80c53b0 con 0x7fccb80cb3f0 2024-03-28T15:10:50.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.343+0000 7fccbec1e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb8154b60 0x7fccb80c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.343+0000 7fccbec1e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb8154b60 0x7fccb80c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49726/0 (socket says 172.21.15.186:49726) 2024-03-28T15:10:50.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.343+0000 7fccbec1e640 1 -- 172.21.15.186:0/2085004243 learned_addr learned my addr 172.21.15.186:0/2085004243 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:50.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.343+0000 7fccbdc1c640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fccb80cb3f0 0x7fccb80b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.343+0000 7fccbe41d640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fccb80cafd0 0x7fccb8149960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.344+0000 7fccbec1e640 1 -- 172.21.15.186:0/2085004243 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fccb80cb3f0 msgr2=0x7fccb80b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.344+0000 7fccbec1e640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fccb80cb3f0 0x7fccb80b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.344+0000 7fccbec1e640 1 -- 172.21.15.186:0/2085004243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fccb80cafd0 msgr2=0x7fccb8149960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.344+0000 7fccbec1e640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fccb80cafd0 0x7fccb8149960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.344+0000 7fccbec1e640 1 -- 172.21.15.186:0/2085004243 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb80c8470 con 0x7fccb8154b60 2024-03-28T15:10:50.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.344+0000 7fccbec1e640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb8154b60 0x7fccb80c93b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fccb00098c0 tx=0x7fccb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.344+0000 7fccaeffd640 1 -- 172.21.15.186:0/2085004243 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0034040 con 0x7fccb8154b60 2024-03-28T15:10:50.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.344+0000 7fccaeffd640 1 -- 172.21.15.186:0/2085004243 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb0030410 con 0x7fccb8154b60 2024-03-28T15:10:50.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.344+0000 7fccc0681640 1 -- 172.21.15.186:0/2085004243 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb814ad10 con 0x7fccb8154b60 2024-03-28T15:10:50.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.344+0000 7fccaeffd640 1 -- 172.21.15.186:0/2085004243 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0039bb0 con 0x7fccb8154b60 2024-03-28T15:10:50.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.344+0000 7fccc0681640 1 -- 172.21.15.186:0/2085004243 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb8149160 con 0x7fccb8154b60 2024-03-28T15:10:50.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.345+0000 7fccc0681640 1 -- 172.21.15.186:0/2085004243 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccb80686d0 con 0x7fccb8154b60 2024-03-28T15:10:50.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.345+0000 7fccaeffd640 1 -- 172.21.15.186:0/2085004243 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fccb0037070 con 0x7fccb8154b60 2024-03-28T15:10:50.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.346+0000 7fccaeffd640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcc9c06c530 0x7fcc9c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.346+0000 7fccbe41d640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcc9c06c530 0x7fcc9c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.346+0000 7fccaeffd640 1 -- 172.21.15.186:0/2085004243 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7fccb00ad6a0 con 0x7fccb8154b60 2024-03-28T15:10:50.352 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.346+0000 7fccbe41d640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcc9c06c530 0x7fcc9c06e9f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fccb4011340 tx=0x7fccb400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.352 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.350+0000 7fccaeffd640 1 -- 172.21.15.186:0/2085004243 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fccb007b600 con 0x7fccb8154b60 2024-03-28T15:10:50.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.525+0000 7fccc0681640 1 -- 172.21.15.186:0/2085004243 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fccb815ab60 con 0x7fcc9c06c530 2024-03-28T15:10:50.526 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.526+0000 7fccaeffd640 1 -- 172.21.15.186:0/2085004243 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fccb815ab60 con 0x7fcc9c06c530 2024-03-28T15:10:50.526 INFO:teuthology.orchestra.run.smithi186.stderr:instructed osd(s) 6 to deep-scrub 2024-03-28T15:10:50.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.527+0000 7fccc0681640 1 -- 172.21.15.186:0/2085004243 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcc9c06c530 msgr2=0x7fcc9c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.527+0000 7fccc0681640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcc9c06c530 0x7fcc9c06e9f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fccb4011340 tx=0x7fccb400b040 comp rx=0 tx=0).stop 2024-03-28T15:10:50.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.527+0000 7fccc0681640 1 -- 172.21.15.186:0/2085004243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb8154b60 msgr2=0x7fccb80c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.527+0000 7fccc0681640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb8154b60 0x7fccb80c93b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fccb00098c0 tx=0x7fccb00046c0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.528+0000 7fccc0681640 1 -- 172.21.15.186:0/2085004243 shutdown_connections 2024-03-28T15:10:50.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.528+0000 7fccc0681640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fcc9c06c530 0x7fcc9c06e9f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.528+0000 7fccc0681640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb8154b60 0x7fccb80c93b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.528+0000 7fccc0681640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fccb80cb3f0 0x7fccb80b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.528+0000 7fccc0681640 1 --2- 172.21.15.186:0/2085004243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fccb80cafd0 0x7fccb8149960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.528+0000 7fccc0681640 1 -- 172.21.15.186:0/2085004243 >> 172.21.15.186:0/2085004243 conn(0x7fccb8073b80 msgr2=0x7fccb80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:50.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.528+0000 7fccc0681640 1 -- 172.21.15.186:0/2085004243 shutdown_connections 2024-03-28T15:10:50.530 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.528+0000 7fccc0681640 1 -- 172.21.15.186:0/2085004243 wait complete. 2024-03-28T15:10:50.539 INFO:tasks.ceph:Scrubbing osd.7 2024-03-28T15:10:50.540 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-28T15:10:50.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.644+0000 7f5a1ba40640 1 Processor -- start 2024-03-28T15:10:50.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.644+0000 7f5a1ba40640 1 -- start start 2024-03-28T15:10:50.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a1ba40640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5a140c12a0 0x7f5a140712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a1ba40640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a140b1d30 0x7f5a140717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a1ba40640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a140c17e0 con 0x7f5a141515b0 2024-03-28T15:10:50.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a1ba40640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f5a14151990 con 0x7f5a140c12a0 2024-03-28T15:10:50.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a1ba40640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5a1406d130 con 0x7f5a140b1d30 2024-03-28T15:10:50.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a18fdb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5a140c12a0 0x7f5a140712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a18fdb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5a140c12a0 0x7f5a140712b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49730/0 (socket says 172.21.15.186:49730) 2024-03-28T15:10:50.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a18fdb640 1 -- 172.21.15.186:0/3304669871 learned_addr learned my addr 172.21.15.186:0/3304669871 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:50.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a19fdd640 1 --2- 172.21.15.186:0/3304669871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a140b1d30 0x7f5a140717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a18fdb640 1 -- 172.21.15.186:0/3304669871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a140b1d30 msgr2=0x7f5a140717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:50.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a18fdb640 1 --2- 172.21.15.186:0/3304669871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a140b1d30 0x7f5a140717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a18fdb640 1 -- 172.21.15.186:0/3304669871 >> v1:172.21.15.191:6789/0 conn(0x7f5a141515b0 legacy=0x7f5a14157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.645+0000 7f5a18fdb640 1 -- 172.21.15.186:0/3304669871 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1406cb70 con 0x7f5a140c12a0 2024-03-28T15:10:50.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.646+0000 7f5a18fdb640 1 --2- 172.21.15.186:0/3304669871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140c12a0 0x7f5a140712b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400bb10 tx=0x7f5a0402e1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d5f4223fc9554c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.646+0000 7f5a0bfff640 1 -- 172.21.15.186:0/3304669871 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0403b070 con 0x7f5a140c12a0 2024-03-28T15:10:50.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.646+0000 7f5a0bfff640 1 -- 172.21.15.186:0/3304669871 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a04036440 con 0x7f5a140c12a0 2024-03-28T15:10:50.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.646+0000 7f5a0bfff640 1 -- 172.21.15.186:0/3304669871 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04036810 con 0x7f5a140c12a0 2024-03-28T15:10:50.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.647+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3304669871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140c12a0 msgr2=0x7f5a140712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.647+0000 7f5a1ba40640 1 --2- 172.21.15.186:0/3304669871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140c12a0 0x7f5a140712b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400bb10 tx=0x7f5a0402e1f0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.647+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3304669871 shutdown_connections 2024-03-28T15:10:50.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.647+0000 7f5a1ba40640 1 --2- 172.21.15.186:0/3304669871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a140b1d30 0x7f5a140717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.647+0000 7f5a1ba40640 1 --2- 172.21.15.186:0/3304669871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140c12a0 0x7f5a140712b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.647+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3304669871 >> 172.21.15.186:0/3304669871 conn(0x7f5a140c8a20 msgr2=0x7f5a140c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:50.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.647+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3304669871 shutdown_connections 2024-03-28T15:10:50.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.647+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3304669871 wait complete. 2024-03-28T15:10:50.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.648+0000 7f5a1ba40640 1 Processor -- start 2024-03-28T15:10:50.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.648+0000 7f5a1ba40640 1 -- start start 2024-03-28T15:10:50.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a1ba40640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140b1d30 0x7f5a141f0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a1ba40640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a140c12a0 0x7f5a14144b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a1ba40640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a141515b0 0x7f5a1414f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a1ba40640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5a14072d20 con 0x7f5a140b1d30 2024-03-28T15:10:50.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a1ba40640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1406c920 con 0x7f5a141515b0 2024-03-28T15:10:50.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a1ba40640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f5a14158250 con 0x7f5a140c12a0 2024-03-28T15:10:50.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a197dc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140b1d30 0x7f5a141f0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a197dc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140b1d30 0x7f5a141f0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49740/0 (socket says 172.21.15.186:49740) 2024-03-28T15:10:50.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a197dc640 1 -- 172.21.15.186:0/3345586307 learned_addr learned my addr 172.21.15.186:0/3345586307 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:50.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a19fdd640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a141515b0 0x7f5a1414f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a18fdb640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a140c12a0 0x7f5a14144b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a197dc640 1 -- 172.21.15.186:0/3345586307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a140c12a0 msgr2=0x7f5a14144b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a197dc640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a140c12a0 0x7f5a14144b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a197dc640 1 -- 172.21.15.186:0/3345586307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a141515b0 msgr2=0x7f5a1414f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a197dc640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a141515b0 0x7f5a1414f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a197dc640 1 -- 172.21.15.186:0/3345586307 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a14158860 con 0x7f5a140b1d30 2024-03-28T15:10:50.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a197dc640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140b1d30 0x7f5a141f0f90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000c3f0 tx=0x7f5a1000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a09ffb640 1 -- 172.21.15.186:0/3345586307 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a100052d0 con 0x7f5a140b1d30 2024-03-28T15:10:50.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a09ffb640 1 -- 172.21.15.186:0/3345586307 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a10005490 con 0x7f5a140b1d30 2024-03-28T15:10:50.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3345586307 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a14158b40 con 0x7f5a140b1d30 2024-03-28T15:10:50.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.649+0000 7f5a09ffb640 1 -- 172.21.15.186:0/3345586307 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a10007260 con 0x7f5a140b1d30 2024-03-28T15:10:50.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.650+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3345586307 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a14159100 con 0x7f5a140b1d30 2024-03-28T15:10:50.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.650+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3345586307 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5a14158610 con 0x7f5a140b1d30 2024-03-28T15:10:50.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.651+0000 7f5a09ffb640 1 -- 172.21.15.186:0/3345586307 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5a10007400 con 0x7f5a140b1d30 2024-03-28T15:10:50.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.651+0000 7f5a09ffb640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f59f806c530 0x7f59f806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.651+0000 7f5a09ffb640 1 -- 172.21.15.186:0/3345586307 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f5a1008acd0 con 0x7f5a140b1d30 2024-03-28T15:10:50.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.651+0000 7f5a18fdb640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f59f806c530 0x7f59f806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.651+0000 7f5a09ffb640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] conn(0x7f59f8074bd0 0x7f59f8077050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.651+0000 7f5a09ffb640 1 -- 172.21.15.186:0/3345586307 --> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a10009e50 con 0x7f59f8074bd0 2024-03-28T15:10:50.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.651+0000 7f5a09ffb640 1 -- 172.21.15.186:0/3345586307 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5a1008d3d0 con 0x7f5a140b1d30 2024-03-28T15:10:50.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.652+0000 7f5a19fdd640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] conn(0x7f59f8074bd0 0x7f59f8077050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.652+0000 7f5a18fdb640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f59f806c530 0x7f59f806e9f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5a04038000 tx=0x7f5a04008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.652+0000 7f5a19fdd640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] conn(0x7f59f8074bd0 0x7f59f8077050 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.653+0000 7f5a09ffb640 1 -- 172.21.15.186:0/3345586307 <== osd.7 v2:172.21.15.191:6816/562993402 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5a10009e50 con 0x7f59f8074bd0 2024-03-28T15:10:50.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.665+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3345586307 --> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5a14069570 con 0x7f59f8074bd0 2024-03-28T15:10:50.666 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a09ffb640 1 -- 172.21.15.186:0/3345586307 <== osd.7 v2:172.21.15.191:6816/562993402 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5a14069570 con 0x7f59f8074bd0 2024-03-28T15:10:50.666 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-28T15:10:50.667 INFO:teuthology.orchestra.run.smithi186.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T15:10:50.667 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-28T15:10:50.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3345586307 >> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] conn(0x7f59f8074bd0 msgr2=0x7f59f8077050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a1ba40640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.191:6816/562993402,v1:172.21.15.191:6817/562993402] conn(0x7f59f8074bd0 0x7f59f8077050 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3345586307 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f59f806c530 msgr2=0x7f59f806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a1ba40640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f59f806c530 0x7f59f806e9f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5a04038000 tx=0x7f5a04008040 comp rx=0 tx=0).stop 2024-03-28T15:10:50.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3345586307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140b1d30 msgr2=0x7f5a141f0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a1ba40640 1 --2- 172.21.15.186:0/3345586307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140b1d30 0x7f5a141f0f90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000c3f0 tx=0x7f5a1000c8b0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a19fdd640 1 -- 172.21.15.186:0/3345586307 reap_dead start 2024-03-28T15:10:50.668 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3345586307 shutdown_connections 2024-03-28T15:10:50.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3345586307 >> 172.21.15.186:0/3345586307 conn(0x7f5a140c8a20 msgr2=0x7f5a140c6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:50.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3345586307 shutdown_connections 2024-03-28T15:10:50.669 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.666+0000 7f5a1ba40640 1 -- 172.21.15.186:0/3345586307 wait complete. 2024-03-28T15:10:50.677 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-28T15:10:50.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.782+0000 7f8348876640 1 Processor -- start 2024-03-28T15:10:50.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.782+0000 7f8348876640 1 -- start start 2024-03-28T15:10:50.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8348876640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8344151870 0x7f83440c7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8348876640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f834406db50 0x7f834406e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8348876640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83440c8600 con 0x7f83440c8110 2024-03-28T15:10:50.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8348876640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f8344158070 con 0x7f834406db50 2024-03-28T15:10:50.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8348876640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f8344157740 con 0x7f8344151870 2024-03-28T15:10:50.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f833bfff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f83440c8110 0x7f834406ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:60486/0 (socket says 172.21.15.186:60486) 2024-03-28T15:10:50.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f833bfff640 1 -- 172.21.15.186:0/1802192400 learned_addr learned my addr 172.21.15.186:0/1802192400 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:10:50.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f834259c640 1 --2- 172.21.15.186:0/1802192400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8344151870 0x7f83440c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.787 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8342d9d640 1 --2- 172.21.15.186:0/1802192400 >> v2:172.21.15.191:3300/0 conn(0x7f834406db50 0x7f834406e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.787 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8341d9b640 1 -- 172.21.15.186:0/1802192400 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 796665850 0 0) 0x7f83440c8600 con 0x7f83440c8110 2024-03-28T15:10:50.787 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8341d9b640 1 -- 172.21.15.186:0/1802192400 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8330005730 con 0x7f83440c8110 2024-03-28T15:10:50.787 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8342d9d640 1 -- 172.21.15.186:0/1802192400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8344151870 msgr2=0x7f83440c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:50.787 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8342d9d640 1 --2- 172.21.15.186:0/1802192400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8344151870 0x7f83440c7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.787 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8342d9d640 1 -- 172.21.15.186:0/1802192400 >> v1:172.21.15.186:6789/0 conn(0x7f83440c8110 legacy=0x7f834406ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.787 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f8342d9d640 1 -- 172.21.15.186:0/1802192400 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8344157a20 con 0x7f834406db50 2024-03-28T15:10:50.787 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.783+0000 7f834259c640 1 --2- 172.21.15.186:0/1802192400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8344151870 0x7f83440c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:50.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.784+0000 7f8342d9d640 1 --2- 172.21.15.186:0/1802192400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834406db50 0x7f834406e000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f833c009d90 tx=0x7f833c02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66b0fa503f40fcb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.784+0000 7f8341d9b640 1 -- 172.21.15.186:0/1802192400 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c039070 con 0x7f834406db50 2024-03-28T15:10:50.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.784+0000 7f8341d9b640 1 -- 172.21.15.186:0/1802192400 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833c02cce0 con 0x7f834406db50 2024-03-28T15:10:50.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.784+0000 7f8341d9b640 1 -- 172.21.15.186:0/1802192400 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c034640 con 0x7f834406db50 2024-03-28T15:10:50.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.785+0000 7f8348876640 1 -- 172.21.15.186:0/1802192400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834406db50 msgr2=0x7f834406e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.785+0000 7f8348876640 1 --2- 172.21.15.186:0/1802192400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834406db50 0x7f834406e000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f833c009d90 tx=0x7f833c02bf30 comp rx=0 tx=0).stop 2024-03-28T15:10:50.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.785+0000 7f8348876640 1 -- 172.21.15.186:0/1802192400 shutdown_connections 2024-03-28T15:10:50.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.785+0000 7f8348876640 1 --2- 172.21.15.186:0/1802192400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834406db50 0x7f834406e000 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.785+0000 7f8348876640 1 --2- 172.21.15.186:0/1802192400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8344151870 0x7f83440c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.785+0000 7f8348876640 1 -- 172.21.15.186:0/1802192400 >> 172.21.15.186:0/1802192400 conn(0x7f834406c010 msgr2=0x7f83440cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:50.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.785+0000 7f8348876640 1 -- 172.21.15.186:0/1802192400 shutdown_connections 2024-03-28T15:10:50.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.785+0000 7f8348876640 1 -- 172.21.15.186:0/1802192400 wait complete. 2024-03-28T15:10:50.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.786+0000 7f8348876640 1 Processor -- start 2024-03-28T15:10:50.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.786+0000 7f8348876640 1 -- start start 2024-03-28T15:10:50.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f8348876640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f834406db50 0x7f83441475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f8348876640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83440c8110 0x7f8344155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f8348876640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8344151870 0x7f834414f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f8348876640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83440c78d0 con 0x7f834406db50 2024-03-28T15:10:50.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f8348876640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8344151c50 con 0x7f8344151870 2024-03-28T15:10:50.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f8348876640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f834406f2f0 con 0x7f83440c8110 2024-03-28T15:10:50.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f834259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f834406db50 0x7f83441475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f834259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f834406db50 0x7f83441475f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49748/0 (socket says 172.21.15.186:49748) 2024-03-28T15:10:50.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f834259c640 1 -- 172.21.15.186:0/3510932463 learned_addr learned my addr 172.21.15.186:0/3510932463 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:50.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f833bfff640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83440c8110 0x7f8344155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f8342d9d640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8344151870 0x7f834414f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f834259c640 1 -- 172.21.15.186:0/3510932463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83440c8110 msgr2=0x7f8344155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f834259c640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83440c8110 0x7f8344155e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f834259c640 1 -- 172.21.15.186:0/3510932463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8344151870 msgr2=0x7f834414f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:50.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f834259c640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8344151870 0x7f834414f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:50.794 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f834259c640 1 -- 172.21.15.186:0/3510932463 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83441ec450 con 0x7f834406db50 2024-03-28T15:10:50.794 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f834259c640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f834406db50 0x7f83441475f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f833400ebe0 tx=0x7f833400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.794 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f833affd640 1 -- 172.21.15.186:0/3510932463 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334017070 con 0x7f834406db50 2024-03-28T15:10:50.794 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f833affd640 1 -- 172.21.15.186:0/3510932463 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833400cc30 con 0x7f834406db50 2024-03-28T15:10:50.794 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f8348876640 1 -- 172.21.15.186:0/3510932463 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83441ec6f0 con 0x7f834406db50 2024-03-28T15:10:50.794 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.787+0000 7f833affd640 1 -- 172.21.15.186:0/3510932463 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334013470 con 0x7f834406db50 2024-03-28T15:10:50.794 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.788+0000 7f8348876640 1 -- 172.21.15.186:0/3510932463 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83441eccb0 con 0x7f834406db50 2024-03-28T15:10:50.794 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.789+0000 7f8348876640 1 -- 172.21.15.186:0/3510932463 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8344069570 con 0x7f834406db50 2024-03-28T15:10:50.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.789+0000 7f833affd640 1 -- 172.21.15.186:0/3510932463 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f833401a070 con 0x7f834406db50 2024-03-28T15:10:50.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.789+0000 7f833affd640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f832406c530 0x7f832406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:50.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.790+0000 7f833affd640 1 -- 172.21.15.186:0/3510932463 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f833408b8a0 con 0x7f834406db50 2024-03-28T15:10:50.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.790+0000 7f833bfff640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f832406c530 0x7f832406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:50.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.790+0000 7f833bfff640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f832406c530 0x7f832406e9f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f83441f0890 tx=0x7f832c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:50.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:50.794+0000 7f833affd640 1 -- 172.21.15.186:0/3510932463 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f83340595c0 con 0x7f834406db50 2024-03-28T15:10:51.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.021+0000 7f8348876640 1 -- 172.21.15.186:0/3510932463 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f8344143200 con 0x7f832406c530 2024-03-28T15:10:51.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.021+0000 7f833affd640 1 -- 172.21.15.186:0/3510932463 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f832c00f820 con 0x7f832406c530 2024-03-28T15:10:51.022 INFO:teuthology.orchestra.run.smithi186.stderr:instructed osd(s) 7 to deep-scrub 2024-03-28T15:10:51.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 -- 172.21.15.186:0/3510932463 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f832406c530 msgr2=0x7f832406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:51.024 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f832406c530 0x7f832406e9f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f83441f0890 tx=0x7f832c009040 comp rx=0 tx=0).stop 2024-03-28T15:10:51.024 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 -- 172.21.15.186:0/3510932463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f834406db50 msgr2=0x7f83441475f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:51.024 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f834406db50 0x7f83441475f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f833400ebe0 tx=0x7f833400c520 comp rx=0 tx=0).stop 2024-03-28T15:10:51.024 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 -- 172.21.15.186:0/3510932463 shutdown_connections 2024-03-28T15:10:51.024 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f832406c530 0x7f832406e9f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.024 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8344151870 0x7f834414f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.024 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83440c8110 0x7f8344155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.025 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 --2- 172.21.15.186:0/3510932463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f834406db50 0x7f83441475f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.025 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 -- 172.21.15.186:0/3510932463 >> 172.21.15.186:0/3510932463 conn(0x7f834406c010 msgr2=0x7f83440c7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:51.025 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 -- 172.21.15.186:0/3510932463 shutdown_connections 2024-03-28T15:10:51.025 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.023+0000 7f8348876640 1 -- 172.21.15.186:0/3510932463 wait complete. 2024-03-28T15:10:51.035 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:10:51.147 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.147+0000 7ff724513640 1 Processor -- start 2024-03-28T15:10:51.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.147+0000 7ff724513640 1 -- start start 2024-03-28T15:10:51.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff724513640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff71c06d670 0x7ff71c06f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:51.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff724513640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff71c06da50 0x7ff71c06dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:51.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff724513640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff71c06f770 con 0x7ff71c06fb90 2024-03-28T15:10:51.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff724513640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff71c0c91a0 con 0x7ff71c06da50 2024-03-28T15:10:51.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff724513640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7ff71c0c4c60 con 0x7ff71c06d670 2024-03-28T15:10:51.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff722ab0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff71c06da50 0x7ff71c06dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:51.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff722ab0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff71c06da50 0x7ff71c06dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49758/0 (socket says 172.21.15.186:49758) 2024-03-28T15:10:51.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff721aae640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff71c06fb90 0x7ff71c0c8a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:60498/0 (socket says 172.21.15.186:60498) 2024-03-28T15:10:51.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff722ab0640 1 -- 172.21.15.186:0/2699437588 learned_addr learned my addr 172.21.15.186:0/2699437588 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:51.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff7222af640 1 --2- 172.21.15.186:0/2699437588 >> v2:172.21.15.191:3300/0 conn(0x7ff71c06d670 0x7ff71c06f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:51.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff722ab0640 1 -- 172.21.15.186:0/2699437588 >> v2:172.21.15.191:3300/0 conn(0x7ff71c06d670 msgr2=0x7ff71c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:10:51.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff722ab0640 1 --2- 172.21.15.186:0/2699437588 >> v2:172.21.15.191:3300/0 conn(0x7ff71c06d670 0x7ff71c06f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff722ab0640 1 -- 172.21.15.186:0/2699437588 >> v1:172.21.15.186:6789/0 conn(0x7ff71c06fb90 legacy=0x7ff71c0c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:51.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff722ab0640 1 -- 172.21.15.186:0/2699437588 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c159500 con 0x7ff71c06da50 2024-03-28T15:10:51.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.148+0000 7ff7212ad640 1 -- 172.21.15.186:0/2699437588 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3869559476 0 0) 0x7ff71c06f770 con 0x7ff71c06fb90 2024-03-28T15:10:51.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.149+0000 7ff722ab0640 1 --2- 172.21.15.186:0/2699437588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff71c06da50 0x7ff71c06dee0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0cf120 tx=0x7ff71402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2674db8eea224f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:51.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.149+0000 7ff7212ad640 1 -- 172.21.15.186:0/2699437588 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71402ca80 con 0x7ff71c06da50 2024-03-28T15:10:51.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.149+0000 7ff7212ad640 1 -- 172.21.15.186:0/2699437588 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff71402cc20 con 0x7ff71c06da50 2024-03-28T15:10:51.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.149+0000 7ff7212ad640 1 -- 172.21.15.186:0/2699437588 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7140343f0 con 0x7ff71c06da50 2024-03-28T15:10:51.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.150+0000 7ff724513640 1 -- 172.21.15.186:0/2699437588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff71c06da50 msgr2=0x7ff71c06dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:51.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.150+0000 7ff724513640 1 --2- 172.21.15.186:0/2699437588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff71c06da50 0x7ff71c06dee0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0cf120 tx=0x7ff71402bee0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.150+0000 7ff724513640 1 -- 172.21.15.186:0/2699437588 shutdown_connections 2024-03-28T15:10:51.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.150+0000 7ff724513640 1 --2- 172.21.15.186:0/2699437588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff71c06da50 0x7ff71c06dee0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.150+0000 7ff724513640 1 --2- 172.21.15.186:0/2699437588 >> v2:172.21.15.191:3300/0 conn(0x7ff71c06d670 0x7ff71c06f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.150+0000 7ff724513640 1 -- 172.21.15.186:0/2699437588 >> 172.21.15.186:0/2699437588 conn(0x7ff71c067670 msgr2=0x7ff71c071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:51.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.150+0000 7ff724513640 1 -- 172.21.15.186:0/2699437588 shutdown_connections 2024-03-28T15:10:51.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.150+0000 7ff724513640 1 -- 172.21.15.186:0/2699437588 wait complete. 2024-03-28T15:10:51.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.152+0000 7ff724513640 1 Processor -- start 2024-03-28T15:10:51.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.152+0000 7ff724513640 1 -- start start 2024-03-28T15:10:51.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.152+0000 7ff724513640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff71c06d670 0x7ff71c156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:51.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.152+0000 7ff724513640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff71c06da50 0x7ff71c157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:51.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.152+0000 7ff724513640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff71c06fb90 0x7ff71c158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:51.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.152+0000 7ff724513640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c0c34c0 con 0x7ff71c06fb90 2024-03-28T15:10:51.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff724513640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c0c31e0 con 0x7ff71c06da50 2024-03-28T15:10:51.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff724513640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7ff71c0c50f0 con 0x7ff71c06d670 2024-03-28T15:10:51.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff722ab0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff71c06fb90 0x7ff71c158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:51.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff7222af640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff71c06d670 0x7ff71c156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:51.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff721aae640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff71c06da50 0x7ff71c157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:51.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff722ab0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff71c06fb90 0x7ff71c158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:49774/0 (socket says 172.21.15.186:49774) 2024-03-28T15:10:51.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff7222af640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff71c06d670 0x7ff71c156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:49038/0 (socket says 172.21.15.186:49038) 2024-03-28T15:10:51.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff722ab0640 1 -- 172.21.15.186:0/3967795702 learned_addr learned my addr 172.21.15.186:0/3967795702 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:10:51.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff722ab0640 1 -- 172.21.15.186:0/3967795702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff71c06d670 msgr2=0x7ff71c156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:51.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff722ab0640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff71c06d670 0x7ff71c156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff722ab0640 1 -- 172.21.15.186:0/3967795702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff71c06da50 msgr2=0x7ff71c157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:51.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff722ab0640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff71c06da50 0x7ff71c157e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff722ab0640 1 -- 172.21.15.186:0/3967795702 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c06c8e0 con 0x7ff71c06fb90 2024-03-28T15:10:51.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff7222af640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff71c06d670 0x7ff71c156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:51.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff721aae640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff71c06da50 0x7ff71c157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:10:51.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.153+0000 7ff722ab0640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff71c06fb90 0x7ff71c158350 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff714009f40 tx=0x7ff714004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:51.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.154+0000 7ff712ffd640 1 -- 172.21.15.186:0/3967795702 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714043020 con 0x7ff71c06fb90 2024-03-28T15:10:51.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.154+0000 7ff724513640 1 -- 172.21.15.186:0/3967795702 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff71c06c1c0 con 0x7ff71c06fb90 2024-03-28T15:10:51.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.154+0000 7ff724513640 1 -- 172.21.15.186:0/3967795702 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff71c1eccc0 con 0x7ff71c06fb90 2024-03-28T15:10:51.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.154+0000 7ff712ffd640 1 -- 172.21.15.186:0/3967795702 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff71403e450 con 0x7ff71c06fb90 2024-03-28T15:10:51.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.154+0000 7ff712ffd640 1 -- 172.21.15.186:0/3967795702 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714009510 con 0x7ff71c06fb90 2024-03-28T15:10:51.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.155+0000 7ff724513640 1 -- 172.21.15.186:0/3967795702 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff71c0686d0 con 0x7ff71c06fb90 2024-03-28T15:10:51.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.156+0000 7ff712ffd640 1 -- 172.21.15.186:0/3967795702 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff714046020 con 0x7ff71c06fb90 2024-03-28T15:10:51.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.156+0000 7ff712ffd640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff70006c530 0x7ff70006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:10:51.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.156+0000 7ff712ffd640 1 -- 172.21.15.186:0/3967795702 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7ff7140adfc0 con 0x7ff71c06fb90 2024-03-28T15:10:51.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.156+0000 7ff7222af640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff70006c530 0x7ff70006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:10:51.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.157+0000 7ff7222af640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff70006c530 0x7ff70006e9f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff718009e90 tx=0x7ff718009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:10:51.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.162+0000 7ff712ffd640 1 -- 172.21.15.186:0/3967795702 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7ff71404d030 con 0x7ff71c06fb90 2024-03-28T15:10:51.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.347+0000 7ff724513640 1 -- 172.21.15.186:0/3967795702 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff71c0c9ac0 con 0x7ff70006c530 2024-03-28T15:10:51.353 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.352+0000 7ff712ffd640 1 -- 172.21.15.186:0/3967795702 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254651 (secure 0 0 0) 0x7ff71c0c9ac0 con 0x7ff70006c530 2024-03-28T15:10:51.353 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:10:51.357 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:10:51.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 -- 172.21.15.186:0/3967795702 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff70006c530 msgr2=0x7ff70006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:51.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff70006c530 0x7ff70006e9f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff718009e90 tx=0x7ff718009190 comp rx=0 tx=0).stop 2024-03-28T15:10:51.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 -- 172.21.15.186:0/3967795702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff71c06fb90 msgr2=0x7ff71c158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:10:51.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff71c06fb90 0x7ff71c158350 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff714009f40 tx=0x7ff714004a40 comp rx=0 tx=0).stop 2024-03-28T15:10:51.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 -- 172.21.15.186:0/3967795702 shutdown_connections 2024-03-28T15:10:51.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7ff70006c530 0x7ff70006e9f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff71c06fb90 0x7ff71c158350 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff71c06da50 0x7ff71c157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 --2- 172.21.15.186:0/3967795702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff71c06d670 0x7ff71c156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:10:51.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 -- 172.21.15.186:0/3967795702 >> 172.21.15.186:0/3967795702 conn(0x7ff71c067670 msgr2=0x7ff71c151740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:10:51.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 -- 172.21.15.186:0/3967795702 shutdown_connections 2024-03-28T15:10:51.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:10:51.356+0000 7ff724513640 1 -- 172.21.15.186:0/3967795702 wait complete. 2024-03-28T15:10:51.367 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-03-28T15:10:50.751739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1123980267,"num_objects":10721,"num_object_clones":0,"num_object_copies":21442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10721,"num_whiteouts":0,"num_read":31288,"num_read_kb":1091324,"num_write":28285,"num_write_kb":1250743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":94136,"num_omap_keys":184,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28587,"ondisk_log_size":28587,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2684552,"kb_used_data":2250632,"kb_used_omap":12,"kb_used_meta":433843,"kb_avail":752290168,"statfs":{"total":773094113280,"available":770345132032,"internally_reserved":0,"allocated":2304647168,"data_stored":2247513334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":444255793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3477308,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":155,"num_write_kb":3474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":94136,"num_omap_keys":184,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002968"},"pg_stats":[{"pgid":"3.3f","version":"37'335","reported_seq":3248,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.950296+0000","last_change":"2024-03-28T15:04:56.745680+0000","last_active":"2024-03-28T15:10:38.950296+0000","last_peered":"2024-03-28T15:10:38.950296+0000","last_clean":"2024-03-28T15:10:38.950296+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:10:38.950296+0000","last_undegraded":"2024-03-28T15:10:38.950296+0000","last_fullsized":"2024-03-28T15:10:38.950296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:21:41.971527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024885649999999999,"stat_sum":{"num_bytes":26509954,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":839,"num_read_kb":12103,"num_write":327,"num_write_kb":26054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"37'316","reported_seq":2407,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.931358+0000","last_change":"2024-03-28T15:04:56.294141+0000","last_active":"2024-03-28T15:10:38.931358+0000","last_peered":"2024-03-28T15:10:38.931358+0000","last_clean":"2024-03-28T15:10:38.931358+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:10:38.931358+0000","last_undegraded":"2024-03-28T15:10:38.931358+0000","last_fullsized":"2024-03-28T15:10:38.931358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019334860000000001,"stat_sum":{"num_bytes":1345083,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":314,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"37'329","reported_seq":3089,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.923169+0000","last_change":"2024-03-28T15:04:56.198079+0000","last_active":"2024-03-28T15:10:38.923169+0000","last_peered":"2024-03-28T15:10:38.923169+0000","last_clean":"2024-03-28T15:10:38.923169+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:10:38.923169+0000","last_undegraded":"2024-03-28T15:10:38.923169+0000","last_fullsized":"2024-03-28T15:10:38.923169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025151280000000002,"stat_sum":{"num_bytes":12619399,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":685,"num_read_kb":17683,"num_write":321,"num_write_kb":20675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"37'314","reported_seq":3078,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938334+0000","last_change":"2024-03-28T15:04:56.747212+0000","last_active":"2024-03-28T15:10:38.938334+0000","last_peered":"2024-03-28T15:10:38.938334+0000","last_clean":"2024-03-28T15:10:38.938334+0000","last_became_active":"2024-03-28T15:02:12.485356+0000","last_became_peered":"2024-03-28T15:02:12.485356+0000","last_unstale":"2024-03-28T15:10:38.938334+0000","last_undegraded":"2024-03-28T15:10:38.938334+0000","last_fullsized":"2024-03-28T15:10:38.938334+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014203849999999999,"stat_sum":{"num_bytes":19355212,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":695,"num_read_kb":14278,"num_write":306,"num_write_kb":19012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"37'363","reported_seq":2551,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938805+0000","last_change":"2024-03-28T15:04:56.745133+0000","last_active":"2024-03-28T15:10:38.938805+0000","last_peered":"2024-03-28T15:10:38.938805+0000","last_clean":"2024-03-28T15:10:38.938805+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:10:38.938805+0000","last_undegraded":"2024-03-28T15:10:38.938805+0000","last_fullsized":"2024-03-28T15:10:38.938805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:21.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002796064,"stat_sum":{"num_bytes":10679980,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":115,"num_read_kb":4114,"num_write":361,"num_write_kb":10554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"37'327","reported_seq":2639,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.946373+0000","last_change":"2024-03-28T15:04:56.200817+0000","last_active":"2024-03-28T15:10:38.946373+0000","last_peered":"2024-03-28T15:10:38.946373+0000","last_clean":"2024-03-28T15:10:38.946373+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:10:38.946373+0000","last_undegraded":"2024-03-28T15:10:38.946373+0000","last_fullsized":"2024-03-28T15:10:38.946373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:03:09.771351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003798718,"stat_sum":{"num_bytes":10128543,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":239,"num_read_kb":4123,"num_write":321,"num_write_kb":10056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"37'354","reported_seq":3455,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.950756+0000","last_change":"2024-03-28T15:04:56.743147+0000","last_active":"2024-03-28T15:10:38.950756+0000","last_peered":"2024-03-28T15:10:38.950756+0000","last_clean":"2024-03-28T15:10:38.950756+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:10:38.950756+0000","last_undegraded":"2024-03-28T15:10:38.950756+0000","last_fullsized":"2024-03-28T15:10:38.950756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:20.151357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001484766,"stat_sum":{"num_bytes":23645872,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":1028,"num_read_kb":17521,"num_write":350,"num_write_kb":23210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"37'334","reported_seq":2945,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960038+0000","last_change":"2024-03-28T15:04:56.208811+0000","last_active":"2024-03-28T15:10:38.960038+0000","last_peered":"2024-03-28T15:10:38.960038+0000","last_clean":"2024-03-28T15:10:38.960038+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:10:38.960038+0000","last_undegraded":"2024-03-28T15:10:38.960038+0000","last_fullsized":"2024-03-28T15:10:38.960038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017148910000000001,"stat_sum":{"num_bytes":19312172,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":537,"num_read_kb":7179,"num_write":326,"num_write_kb":18943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"37'330","reported_seq":2712,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960654+0000","last_change":"2024-03-28T15:04:56.745714+0000","last_active":"2024-03-28T15:10:38.960654+0000","last_peered":"2024-03-28T15:10:38.960654+0000","last_clean":"2024-03-28T15:10:38.960654+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:10:38.960654+0000","last_undegraded":"2024-03-28T15:10:38.960654+0000","last_fullsized":"2024-03-28T15:10:38.960654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:27:32.222447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023275240000000001,"stat_sum":{"num_bytes":6398085,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":309,"num_read_kb":8863,"num_write":320,"num_write_kb":10489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"37'327","reported_seq":3210,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.961682+0000","last_change":"2024-03-28T15:04:56.534451+0000","last_active":"2024-03-28T15:10:38.961682+0000","last_peered":"2024-03-28T15:10:38.961682+0000","last_clean":"2024-03-28T15:10:38.961682+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:10:38.961682+0000","last_undegraded":"2024-03-28T15:10:38.961682+0000","last_fullsized":"2024-03-28T15:10:38.961682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:43.533807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019348060000000001,"stat_sum":{"num_bytes":19157707,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":810,"num_read_kb":17508,"num_write":311,"num_write_kb":18862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"37'303","reported_seq":3011,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.959082+0000","last_change":"2024-03-28T15:04:56.534170+0000","last_active":"2024-03-28T15:10:38.959082+0000","last_peered":"2024-03-28T15:10:38.959082+0000","last_clean":"2024-03-28T15:10:38.959082+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:10:38.959082+0000","last_undegraded":"2024-03-28T15:10:38.959082+0000","last_fullsized":"2024-03-28T15:10:38.959082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:20.495757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018723170000000001,"stat_sum":{"num_bytes":18249049,"num_objects":129,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":635,"num_read_kb":13018,"num_write":297,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"37'296","reported_seq":2603,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.934653+0000","last_change":"2024-03-28T15:04:56.199593+0000","last_active":"2024-03-28T15:10:38.934653+0000","last_peered":"2024-03-28T15:10:38.934653+0000","last_clean":"2024-03-28T15:10:38.934653+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:10:38.934653+0000","last_undegraded":"2024-03-28T15:10:38.934653+0000","last_fullsized":"2024-03-28T15:10:38.934653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:50.527037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013925400000000001,"stat_sum":{"num_bytes":6129603,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":234,"num_read_kb":4757,"num_write":288,"num_write_kb":10236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"37'297","reported_seq":2860,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936148+0000","last_change":"2024-03-28T15:04:56.207750+0000","last_active":"2024-03-28T15:10:38.936148+0000","last_peered":"2024-03-28T15:10:38.936148+0000","last_clean":"2024-03-28T15:10:38.936148+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:10:38.936148+0000","last_undegraded":"2024-03-28T15:10:38.936148+0000","last_fullsized":"2024-03-28T15:10:38.936148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017572180000000001,"stat_sum":{"num_bytes":15029757,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":489,"num_read_kb":12548,"num_write":286,"num_write_kb":23016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"37'341","reported_seq":2614,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932141+0000","last_change":"2024-03-28T15:04:56.197644+0000","last_active":"2024-03-28T15:10:38.932141+0000","last_peered":"2024-03-28T15:10:38.932141+0000","last_clean":"2024-03-28T15:10:38.932141+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:10:38.932141+0000","last_undegraded":"2024-03-28T15:10:38.932141+0000","last_fullsized":"2024-03-28T15:10:38.932141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002253971,"stat_sum":{"num_bytes":6664925,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":199,"num_read_kb":4930,"num_write":335,"num_write_kb":6638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"37'357","reported_seq":3529,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.926854+0000","last_change":"2024-03-28T15:04:56.293889+0000","last_active":"2024-03-28T15:10:38.926854+0000","last_peered":"2024-03-28T15:10:38.926854+0000","last_clean":"2024-03-28T15:10:38.926854+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:10:38.926854+0000","last_undegraded":"2024-03-28T15:10:38.926854+0000","last_fullsized":"2024-03-28T15:10:38.926854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179673,"stat_sum":{"num_bytes":23044368,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":1098,"num_read_kb":19203,"num_write":350,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"37'357","reported_seq":2912,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.945394+0000","last_change":"2024-03-28T15:04:56.200579+0000","last_active":"2024-03-28T15:10:38.945394+0000","last_peered":"2024-03-28T15:10:38.945394+0000","last_clean":"2024-03-28T15:10:38.945394+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:10:38.945394+0000","last_undegraded":"2024-03-28T15:10:38.945394+0000","last_fullsized":"2024-03-28T15:10:38.945394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:30.228263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039634529999999996,"stat_sum":{"num_bytes":16147473,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":482,"num_read_kb":8871,"num_write":348,"num_write_kb":19203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"37'349","reported_seq":2573,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960198+0000","last_change":"2024-03-28T15:04:56.201087+0000","last_active":"2024-03-28T15:10:38.960198+0000","last_peered":"2024-03-28T15:10:38.960198+0000","last_clean":"2024-03-28T15:10:38.960198+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:10:38.960198+0000","last_undegraded":"2024-03-28T15:10:38.960198+0000","last_fullsized":"2024-03-28T15:10:38.960198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:36.687483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027424319999999999,"stat_sum":{"num_bytes":10498072,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":151,"num_read_kb":9005,"num_write":340,"num_write_kb":14498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"37'311","reported_seq":2415,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936394+0000","last_change":"2024-03-28T15:04:56.746916+0000","last_active":"2024-03-28T15:10:38.936394+0000","last_peered":"2024-03-28T15:10:38.936394+0000","last_clean":"2024-03-28T15:10:38.936394+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:10:38.936394+0000","last_undegraded":"2024-03-28T15:10:38.936394+0000","last_fullsized":"2024-03-28T15:10:38.936394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179399,"stat_sum":{"num_bytes":9508598,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":30,"num_read_kb":42,"num_write":300,"num_write_kb":9445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"37'324","reported_seq":2710,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939190+0000","last_change":"2024-03-28T15:04:56.745421+0000","last_active":"2024-03-28T15:10:38.939190+0000","last_peered":"2024-03-28T15:10:38.939190+0000","last_clean":"2024-03-28T15:10:38.939190+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:10:38.939190+0000","last_undegraded":"2024-03-28T15:10:38.939190+0000","last_fullsized":"2024-03-28T15:10:38.939190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:50.382005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030062439999999999,"stat_sum":{"num_bytes":14185939,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":312,"num_read_kb":12813,"num_write":317,"num_write_kb":18119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"37'356","reported_seq":3288,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937246+0000","last_change":"2024-03-28T15:04:56.534664+0000","last_active":"2024-03-28T15:10:38.937246+0000","last_peered":"2024-03-28T15:10:38.937246+0000","last_clean":"2024-03-28T15:10:38.937246+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:10:38.937246+0000","last_undegraded":"2024-03-28T15:10:38.937246+0000","last_fullsized":"2024-03-28T15:10:38.937246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:10.614269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001798584,"stat_sum":{"num_bytes":30036364,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":858,"num_read_kb":15492,"num_write":348,"num_write_kb":37706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"37'342","reported_seq":2725,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957577+0000","last_change":"2024-03-28T15:04:56.198376+0000","last_active":"2024-03-28T15:10:38.957577+0000","last_peered":"2024-03-28T15:10:38.957577+0000","last_clean":"2024-03-28T15:10:38.957577+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:10:38.957577+0000","last_undegraded":"2024-03-28T15:10:38.957577+0000","last_fullsized":"2024-03-28T15:10:38.957577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025216930000000002,"stat_sum":{"num_bytes":10421413,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":308,"num_read_kb":8356,"num_write":331,"num_write_kb":10350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"37'337","reported_seq":2524,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935152+0000","last_change":"2024-03-28T15:04:56.746844+0000","last_active":"2024-03-28T15:10:38.935152+0000","last_peered":"2024-03-28T15:10:38.935152+0000","last_clean":"2024-03-28T15:10:38.935152+0000","last_became_active":"2024-03-28T15:02:12.178196+0000","last_became_peered":"2024-03-28T15:02:12.178196+0000","last_unstale":"2024-03-28T15:10:38.935152+0000","last_undegraded":"2024-03-28T15:10:38.935152+0000","last_fullsized":"2024-03-28T15:10:38.935152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001195878,"stat_sum":{"num_bytes":9431253,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":119,"num_read_kb":4113,"num_write":327,"num_write_kb":13479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"37'331","reported_seq":2999,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939887+0000","last_change":"2024-03-28T15:04:56.207990+0000","last_active":"2024-03-28T15:10:38.939887+0000","last_peered":"2024-03-28T15:10:38.939887+0000","last_clean":"2024-03-28T15:10:38.939887+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:10:38.939887+0000","last_undegraded":"2024-03-28T15:10:38.939887+0000","last_fullsized":"2024-03-28T15:10:38.939887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020004409999999999,"stat_sum":{"num_bytes":16319296,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":593,"num_read_kb":18222,"num_write":323,"num_write_kb":20203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"37'356","reported_seq":2799,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956957+0000","last_change":"2024-03-28T15:04:56.200922+0000","last_active":"2024-03-28T15:10:38.956957+0000","last_peered":"2024-03-28T15:10:38.956957+0000","last_clean":"2024-03-28T15:10:38.956957+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:10:38.956957+0000","last_undegraded":"2024-03-28T15:10:38.956957+0000","last_fullsized":"2024-03-28T15:10:38.956957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:57.578217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031981639999999999,"stat_sum":{"num_bytes":14893759,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":369,"num_read_kb":10097,"num_write":349,"num_write_kb":14727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"37'314","reported_seq":3256,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.933498+0000","last_change":"2024-03-28T15:04:56.207665+0000","last_active":"2024-03-28T15:10:38.933498+0000","last_peered":"2024-03-28T15:10:38.933498+0000","last_clean":"2024-03-28T15:10:38.933498+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:10:38.933498+0000","last_undegraded":"2024-03-28T15:10:38.933498+0000","last_fullsized":"2024-03-28T15:10:38.933498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001813119,"stat_sum":{"num_bytes":18699537,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":867,"num_read_kb":15099,"num_write":310,"num_write_kb":18421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"37'377","reported_seq":2618,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960107+0000","last_change":"2024-03-28T15:04:56.534606+0000","last_active":"2024-03-28T15:10:38.960107+0000","last_peered":"2024-03-28T15:10:38.960107+0000","last_clean":"2024-03-28T15:10:38.960107+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:10:38.960107+0000","last_undegraded":"2024-03-28T15:10:38.960107+0000","last_fullsized":"2024-03-28T15:10:38.960107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:23:24.123040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019272180000000001,"stat_sum":{"num_bytes":11206818,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":167,"num_read_kb":4115,"num_write":363,"num_write_kb":11124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"37'320","reported_seq":2924,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936599+0000","last_change":"2024-03-28T15:04:56.294225+0000","last_active":"2024-03-28T15:10:38.936599+0000","last_peered":"2024-03-28T15:10:38.936599+0000","last_clean":"2024-03-28T15:10:38.936599+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:10:38.936599+0000","last_undegraded":"2024-03-28T15:10:38.936599+0000","last_fullsized":"2024-03-28T15:10:38.936599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019373070000000001,"stat_sum":{"num_bytes":22270707,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":529,"num_read_kb":12598,"num_write":314,"num_write_kb":21911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"37'315","reported_seq":3228,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.943230+0000","last_change":"2024-03-28T15:04:56.208607+0000","last_active":"2024-03-28T15:10:38.943230+0000","last_peered":"2024-03-28T15:10:38.943230+0000","last_clean":"2024-03-28T15:10:38.943230+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:10:38.943230+0000","last_undegraded":"2024-03-28T15:10:38.943230+0000","last_fullsized":"2024-03-28T15:10:38.943230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020980170000000002,"stat_sum":{"num_bytes":17909578,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":838,"num_read_kb":18539,"num_write":311,"num_write_kb":20928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"37'358","reported_seq":3193,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935743+0000","last_change":"2024-03-28T15:04:56.198529+0000","last_active":"2024-03-28T15:10:38.935743+0000","last_peered":"2024-03-28T15:10:38.935743+0000","last_clean":"2024-03-28T15:10:38.935743+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:10:38.935743+0000","last_undegraded":"2024-03-28T15:10:38.935743+0000","last_fullsized":"2024-03-28T15:10:38.935743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026139499999999999,"stat_sum":{"num_bytes":26288347,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":760,"num_read_kb":8154,"num_write":350,"num_write_kb":25854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"37'359","reported_seq":2900,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956962+0000","last_change":"2024-03-28T15:04:56.200966+0000","last_active":"2024-03-28T15:10:38.956962+0000","last_peered":"2024-03-28T15:10:38.956962+0000","last_clean":"2024-03-28T15:10:38.956962+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:10:38.956962+0000","last_undegraded":"2024-03-28T15:10:38.956962+0000","last_fullsized":"2024-03-28T15:10:38.956962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:37.204800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036450829999999999,"stat_sum":{"num_bytes":10387544,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":467,"num_read_kb":8853,"num_write":352,"num_write_kb":10266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"37'386","reported_seq":2969,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.943105+0000","last_change":"2024-03-28T15:04:56.747393+0000","last_active":"2024-03-28T15:10:38.943105+0000","last_peered":"2024-03-28T15:10:38.943105+0000","last_clean":"2024-03-28T15:10:38.943105+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:10:38.943105+0000","last_undegraded":"2024-03-28T15:10:38.943105+0000","last_fullsized":"2024-03-28T15:10:38.943105+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001020962,"stat_sum":{"num_bytes":12939697,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":508,"num_read_kb":8898,"num_write":372,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"37'384","reported_seq":2787,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932311+0000","last_change":"2024-03-28T15:04:56.534204+0000","last_active":"2024-03-28T15:10:38.932311+0000","last_peered":"2024-03-28T15:10:38.932311+0000","last_clean":"2024-03-28T15:10:38.932311+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:10:38.932311+0000","last_undegraded":"2024-03-28T15:10:38.932311+0000","last_fullsized":"2024-03-28T15:10:38.932311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:56:09.141610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019055229999999999,"stat_sum":{"num_bytes":11304826,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":329,"num_read_kb":5287,"num_write":374,"num_write_kb":11236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"37'322","reported_seq":2854,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.921473+0000","last_change":"2024-03-28T15:04:56.196629+0000","last_active":"2024-03-28T15:10:38.921473+0000","last_peered":"2024-03-28T15:10:38.921473+0000","last_clean":"2024-03-28T15:10:38.921473+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:10:38.921473+0000","last_undegraded":"2024-03-28T15:10:38.921473+0000","last_fullsized":"2024-03-28T15:10:38.921473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:51.357070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078031800000000001,"stat_sum":{"num_bytes":14482293,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":459,"num_read_kb":10396,"num_write":315,"num_write_kb":14305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"36'18","reported_seq":75,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.252882+0000","last_change":"2024-03-28T15:01:31.097415+0000","last_active":"2024-03-28T15:10:01.252882+0000","last_peered":"2024-03-28T15:10:01.252882+0000","last_clean":"2024-03-28T15:10:01.252882+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:10:01.252882+0000","last_undegraded":"2024-03-28T15:10:01.252882+0000","last_fullsized":"2024-03-28T15:10:01.252882+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":16384,"num_write":21,"num_write_kb":4230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"37'325","reported_seq":2639,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.935032+0000","last_change":"2024-03-28T15:04:56.198560+0000","last_active":"2024-03-28T15:10:38.935032+0000","last_peered":"2024-03-28T15:10:38.935032+0000","last_clean":"2024-03-28T15:10:38.935032+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:10:38.935032+0000","last_undegraded":"2024-03-28T15:10:38.935032+0000","last_fullsized":"2024-03-28T15:10:38.935032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023690579999999998,"stat_sum":{"num_bytes":11043752,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":240,"num_read_kb":4317,"num_write":317,"num_write_kb":10950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"37'647","reported_seq":711,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.472792+0000","last_change":"2024-03-28T15:01:31.088437+0000","last_active":"2024-03-28T15:10:02.472792+0000","last_peered":"2024-03-28T15:10:02.472792+0000","last_clean":"2024-03-28T15:10:02.472792+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:10:02.472792+0000","last_undegraded":"2024-03-28T15:10:02.472792+0000","last_fullsized":"2024-03-28T15:10:02.472792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":34526,"num_write":642,"num_write_kb":12757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"36'58","reported_seq":113,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.310338+0000","last_change":"2024-03-28T15:01:31.097499+0000","last_active":"2024-03-28T15:10:02.310338+0000","last_peered":"2024-03-28T15:10:02.310338+0000","last_clean":"2024-03-28T15:10:02.310338+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:10:02.310338+0000","last_undegraded":"2024-03-28T15:10:02.310338+0000","last_fullsized":"2024-03-28T15:10:02.310338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":62,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"37'402","reported_seq":3694,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937492+0000","last_change":"2024-03-28T15:04:56.208655+0000","last_active":"2024-03-28T15:10:38.937492+0000","last_peered":"2024-03-28T15:10:38.937492+0000","last_clean":"2024-03-28T15:10:38.937492+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:10:38.937492+0000","last_undegraded":"2024-03-28T15:10:38.937492+0000","last_fullsized":"2024-03-28T15:10:38.937492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017492600000000001,"stat_sum":{"num_bytes":22338843,"num_objects":188,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":1218,"num_read_kb":22454,"num_write":394,"num_write_kb":30214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"37'604","reported_seq":661,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:30.364435+0000","last_change":"2024-03-28T15:01:31.086252+0000","last_active":"2024-03-28T15:10:30.364435+0000","last_peered":"2024-03-28T15:10:30.364435+0000","last_clean":"2024-03-28T15:10:30.364435+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:10:30.364435+0000","last_undegraded":"2024-03-28T15:10:30.364435+0000","last_fullsized":"2024-03-28T15:10:30.364435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":603,"num_write_kb":12492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"37'341","reported_seq":3348,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937632+0000","last_change":"2024-03-28T15:04:56.293936+0000","last_active":"2024-03-28T15:10:38.937632+0000","last_peered":"2024-03-28T15:10:38.937632+0000","last_clean":"2024-03-28T15:10:38.937632+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:10:38.937632+0000","last_undegraded":"2024-03-28T15:10:38.937632+0000","last_fullsized":"2024-03-28T15:10:38.937632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022536280000000001,"stat_sum":{"num_bytes":19647424,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":933,"num_read_kb":22521,"num_write":330,"num_write_kb":23406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"37'5","reported_seq":56,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.246758+0000","last_change":"2024-03-28T15:01:31.092338+0000","last_active":"2024-03-28T15:09:56.246758+0000","last_peered":"2024-03-28T15:09:56.246758+0000","last_clean":"2024-03-28T15:09:56.246758+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:09:56.246758+0000","last_undegraded":"2024-03-28T15:09:56.246758+0000","last_fullsized":"2024-03-28T15:09:56.246758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:27:43.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"37'366","reported_seq":3228,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960208+0000","last_change":"2024-03-28T15:04:56.200271+0000","last_active":"2024-03-28T15:10:38.960208+0000","last_peered":"2024-03-28T15:10:38.960208+0000","last_clean":"2024-03-28T15:10:38.960208+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:10:38.960208+0000","last_undegraded":"2024-03-28T15:10:38.960208+0000","last_fullsized":"2024-03-28T15:10:38.960208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:36:17.765596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043389099999999996,"stat_sum":{"num_bytes":22872414,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":789,"num_read_kb":19564,"num_write":357,"num_write_kb":22493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"37'70","reported_seq":124,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:02.349608+0000","last_change":"2024-03-28T15:01:31.088909+0000","last_active":"2024-03-28T15:10:02.349608+0000","last_peered":"2024-03-28T15:10:02.349608+0000","last_clean":"2024-03-28T15:10:02.349608+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:10:02.349608+0000","last_undegraded":"2024-03-28T15:10:02.349608+0000","last_fullsized":"2024-03-28T15:10:02.349608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":66,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"37'323","reported_seq":2970,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.946940+0000","last_change":"2024-03-28T15:04:56.197873+0000","last_active":"2024-03-28T15:10:38.946940+0000","last_peered":"2024-03-28T15:10:38.946940+0000","last_clean":"2024-03-28T15:10:38.946940+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:10:38.946940+0000","last_undegraded":"2024-03-28T15:10:38.946940+0000","last_fullsized":"2024-03-28T15:10:38.946940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023945189999999999,"stat_sum":{"num_bytes":10186807,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":573,"num_read_kb":8919,"num_write":314,"num_write_kb":10118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"36'193","reported_seq":258,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:23.157756+0000","last_change":"2024-03-28T15:01:31.095935+0000","last_active":"2024-03-28T15:10:23.157756+0000","last_peered":"2024-03-28T15:10:23.157756+0000","last_clean":"2024-03-28T15:10:23.157756+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:10:23.157756+0000","last_undegraded":"2024-03-28T15:10:23.157756+0000","last_fullsized":"2024-03-28T15:10:23.157756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":4122,"num_write":194,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"37'321","reported_seq":3146,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957720+0000","last_change":"2024-03-28T15:04:56.199307+0000","last_active":"2024-03-28T15:10:38.957720+0000","last_peered":"2024-03-28T15:10:38.957720+0000","last_clean":"2024-03-28T15:10:38.957720+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:10:38.957720+0000","last_undegraded":"2024-03-28T15:10:38.957720+0000","last_fullsized":"2024-03-28T15:10:38.957720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:32:03.268809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001254104,"stat_sum":{"num_bytes":14752712,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":752,"num_read_kb":16498,"num_write":310,"num_write_kb":18664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"37'122","reported_seq":201,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.389422+0000","last_change":"2024-03-28T15:01:31.089380+0000","last_active":"2024-03-28T15:10:29.389422+0000","last_peered":"2024-03-28T15:10:29.389422+0000","last_clean":"2024-03-28T15:10:29.389422+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:10:29.389422+0000","last_undegraded":"2024-03-28T15:10:29.389422+0000","last_fullsized":"2024-03-28T15:10:29.389422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.915436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32804,"num_write":115,"num_write_kb":12363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"37'335","reported_seq":2431,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.939280+0000","last_change":"2024-03-28T15:04:56.745455+0000","last_active":"2024-03-28T15:10:38.939280+0000","last_peered":"2024-03-28T15:10:38.939280+0000","last_clean":"2024-03-28T15:10:38.939280+0000","last_became_active":"2024-03-28T15:02:12.182402+0000","last_became_peered":"2024-03-28T15:02:12.182402+0000","last_unstale":"2024-03-28T15:10:38.939280+0000","last_undegraded":"2024-03-28T15:10:38.939280+0000","last_fullsized":"2024-03-28T15:10:38.939280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:32:46.642308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030686839999999999,"stat_sum":{"num_bytes":2787093,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":329,"num_write_kb":2901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"37'101","reported_seq":177,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:48.863146+0000","last_change":"2024-03-28T15:10:48.863213+0000","last_active":"2024-03-28T15:10:48.863146+0000","last_peered":"2024-03-28T15:10:48.863146+0000","last_clean":"2024-03-28T15:10:48.863146+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:10:48.863146+0000","last_undegraded":"2024-03-28T15:10:48.863146+0000","last_fullsized":"2024-03-28T15:10:48.863146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'101","last_scrub_stamp":"2024-03-28T15:10:48.863061+0000","last_deep_scrub":"37'101","last_deep_scrub_stamp":"2024-03-28T15:10:48.863061+0000","last_clean_scrub_stamp":"2024-03-28T15:10:48.863061+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:01:03.258536+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24616,"num_write":94,"num_write_kb":8237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13508,"num_omap_keys":27,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"37'348","reported_seq":3090,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.928128+0000","last_change":"2024-03-28T15:04:56.294429+0000","last_active":"2024-03-28T15:10:38.928128+0000","last_peered":"2024-03-28T15:10:38.928128+0000","last_clean":"2024-03-28T15:10:38.928128+0000","last_became_active":"2024-03-28T15:02:12.180011+0000","last_became_peered":"2024-03-28T15:02:12.180011+0000","last_unstale":"2024-03-28T15:10:38.928128+0000","last_undegraded":"2024-03-28T15:10:38.928128+0000","last_fullsized":"2024-03-28T15:10:38.928128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018715680000000001,"stat_sum":{"num_bytes":20266861,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":674,"num_read_kb":16415,"num_write":337,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":51,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:56.247634+0000","last_change":"2024-03-28T15:01:31.090519+0000","last_active":"2024-03-28T15:09:56.247634+0000","last_peered":"2024-03-28T15:09:56.247634+0000","last_clean":"2024-03-28T15:09:56.247634+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:09:56.247634+0000","last_undegraded":"2024-03-28T15:09:56.247634+0000","last_fullsized":"2024-03-28T15:09:56.247634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"37'358","reported_seq":2567,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.952909+0000","last_change":"2024-03-28T15:04:56.744254+0000","last_active":"2024-03-28T15:10:38.952909+0000","last_peered":"2024-03-28T15:10:38.952909+0000","last_clean":"2024-03-28T15:10:38.952909+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:10:38.952909+0000","last_undegraded":"2024-03-28T15:10:38.952909+0000","last_fullsized":"2024-03-28T15:10:38.952909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:37.876534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022269199999999999,"stat_sum":{"num_bytes":11015620,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":136,"num_read_kb":3039,"num_write":343,"num_write_kb":10934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"37'4","reported_seq":55,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:10.282196+0000","last_change":"2024-03-28T15:01:31.088364+0000","last_active":"2024-03-28T15:10:10.282196+0000","last_peered":"2024-03-28T15:10:10.282196+0000","last_clean":"2024-03-28T15:10:10.282196+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:10:10.282196+0000","last_undegraded":"2024-03-28T15:10:10.282196+0000","last_fullsized":"2024-03-28T15:10:10.282196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"37'344","reported_seq":2987,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.931143+0000","last_change":"2024-03-28T15:04:56.534032+0000","last_active":"2024-03-28T15:10:38.931143+0000","last_peered":"2024-03-28T15:10:38.931143+0000","last_clean":"2024-03-28T15:10:38.931143+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:10:38.931143+0000","last_undegraded":"2024-03-28T15:10:38.931143+0000","last_fullsized":"2024-03-28T15:10:38.931143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:06:03.387822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001897143,"stat_sum":{"num_bytes":17657480,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":571,"num_read_kb":16762,"num_write":332,"num_write_kb":17420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"36'220","reported_seq":301,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:23.127791+0000","last_change":"2024-03-28T15:01:31.094837+0000","last_active":"2024-03-28T15:10:23.127791+0000","last_peered":"2024-03-28T15:10:23.127791+0000","last_clean":"2024-03-28T15:10:23.127791+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:10:23.127791+0000","last_undegraded":"2024-03-28T15:10:23.127791+0000","last_fullsized":"2024-03-28T15:10:23.127791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":83,"num_read_kb":8273,"num_write":219,"num_write_kb":8316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"37'366","reported_seq":2554,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.953014+0000","last_change":"2024-03-28T15:04:56.743437+0000","last_active":"2024-03-28T15:10:38.953014+0000","last_peered":"2024-03-28T15:10:38.953014+0000","last_clean":"2024-03-28T15:10:38.953014+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:10:38.953014+0000","last_undegraded":"2024-03-28T15:10:38.953014+0000","last_fullsized":"2024-03-28T15:10:38.953014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:12:47.009853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001763492,"stat_sum":{"num_bytes":16350020,"num_objects":186,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":115,"num_read_kb":12505,"num_write":360,"num_write_kb":24351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"37'28","reported_seq":88,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:32.560745+0000","last_change":"2024-03-28T15:01:31.092455+0000","last_active":"2024-03-28T15:10:32.560745+0000","last_peered":"2024-03-28T15:10:32.560745+0000","last_clean":"2024-03-28T15:10:32.560745+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:10:32.560745+0000","last_undegraded":"2024-03-28T15:10:32.560745+0000","last_fullsized":"2024-03-28T15:10:32.560745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:04.636216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":29,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"37'290","reported_seq":2751,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.954983+0000","last_change":"2024-03-28T15:04:56.745120+0000","last_active":"2024-03-28T15:10:38.954983+0000","last_peered":"2024-03-28T15:10:38.954983+0000","last_clean":"2024-03-28T15:10:38.954983+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:10:38.954983+0000","last_undegraded":"2024-03-28T15:10:38.954983+0000","last_fullsized":"2024-03-28T15:10:38.954983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:28:19.525504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002782082,"stat_sum":{"num_bytes":11071386,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":387,"num_read_kb":12458,"num_write":285,"num_write_kb":15052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"26'28","reported_seq":91,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:34.525808+0000","last_change":"2024-03-28T15:01:31.092175+0000","last_active":"2024-03-28T15:10:34.525808+0000","last_peered":"2024-03-28T15:10:34.525808+0000","last_clean":"2024-03-28T15:10:34.525808+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:10:34.525808+0000","last_undegraded":"2024-03-28T15:10:34.525808+0000","last_fullsized":"2024-03-28T15:10:34.525808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":28,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"37'337","reported_seq":2700,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957003+0000","last_change":"2024-03-28T15:04:56.745998+0000","last_active":"2024-03-28T15:10:38.957003+0000","last_peered":"2024-03-28T15:10:38.957003+0000","last_clean":"2024-03-28T15:10:38.957003+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:10:38.957003+0000","last_undegraded":"2024-03-28T15:10:38.957003+0000","last_fullsized":"2024-03-28T15:10:38.957003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:36:25.325471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025006389999999998,"stat_sum":{"num_bytes":10266965,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":289,"num_read_kb":13000,"num_write":328,"num_write_kb":14296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"37'120","reported_seq":181,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:34.526187+0000","last_change":"2024-03-28T15:01:31.097901+0000","last_active":"2024-03-28T15:10:34.526187+0000","last_peered":"2024-03-28T15:10:34.526187+0000","last_clean":"2024-03-28T15:10:34.526187+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:10:34.526187+0000","last_undegraded":"2024-03-28T15:10:34.526187+0000","last_fullsized":"2024-03-28T15:10:34.526187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":125,"num_write_kb":4282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"37'362","reported_seq":2665,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957255+0000","last_change":"2024-03-28T15:04:56.200770+0000","last_active":"2024-03-28T15:10:38.957255+0000","last_peered":"2024-03-28T15:10:38.957255+0000","last_clean":"2024-03-28T15:10:38.957255+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:10:38.957255+0000","last_undegraded":"2024-03-28T15:10:38.957255+0000","last_fullsized":"2024-03-28T15:10:38.957255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:09.229020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003899156,"stat_sum":{"num_bytes":10797940,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":229,"num_read_kb":5058,"num_write":352,"num_write_kb":10729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"37'12","reported_seq":73,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:48.967269+0000","last_change":"2024-03-28T15:10:48.967296+0000","last_active":"2024-03-28T15:10:48.967269+0000","last_peered":"2024-03-28T15:10:48.967269+0000","last_clean":"2024-03-28T15:10:48.967269+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:10:48.967269+0000","last_undegraded":"2024-03-28T15:10:48.967269+0000","last_fullsized":"2024-03-28T15:10:48.967269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12","last_scrub_stamp":"2024-03-28T15:10:48.967201+0000","last_deep_scrub":"37'12","last_deep_scrub_stamp":"2024-03-28T15:10:48.967201+0000","last_clean_scrub_stamp":"2024-03-28T15:10:48.967201+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:33:34.173094+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":4136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20754,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"37'326","reported_seq":2660,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.957715+0000","last_change":"2024-03-28T15:04:56.744447+0000","last_active":"2024-03-28T15:10:38.957715+0000","last_peered":"2024-03-28T15:10:38.957715+0000","last_clean":"2024-03-28T15:10:38.957715+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:10:38.957715+0000","last_undegraded":"2024-03-28T15:10:38.957715+0000","last_fullsized":"2024-03-28T15:10:38.957715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:43.786965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026010120000000002,"stat_sum":{"num_bytes":14545241,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":260,"num_read_kb":2397,"num_write":318,"num_write_kb":18466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":208,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:09:57.047088+0000","last_change":"2024-03-28T15:01:20.724163+0000","last_active":"2024-03-28T15:09:57.047088+0000","last_peered":"2024-03-28T15:09:57.047088+0000","last_clean":"2024-03-28T15:09:57.047088+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:09:57.047088+0000","last_undegraded":"2024-03-28T15:09:57.047088+0000","last_fullsized":"2024-03-28T15:09:57.047088+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:19.548090+0000","last_clean_scrub_stamp":"2024-03-28T15:01:19.548090+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:15.791893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"37'84","reported_seq":137,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:03.245313+0000","last_change":"2024-03-28T15:01:31.085453+0000","last_active":"2024-03-28T15:10:03.245313+0000","last_peered":"2024-03-28T15:10:03.245313+0000","last_clean":"2024-03-28T15:10:03.245313+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:10:03.245313+0000","last_undegraded":"2024-03-28T15:10:03.245313+0000","last_fullsized":"2024-03-28T15:10:03.245313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:05.974657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":85,"num_write_kb":4165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"37'326","reported_seq":2490,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.960310+0000","last_change":"2024-03-28T15:04:56.208514+0000","last_active":"2024-03-28T15:10:38.960310+0000","last_peered":"2024-03-28T15:10:38.960310+0000","last_clean":"2024-03-28T15:10:38.960310+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:10:38.960310+0000","last_undegraded":"2024-03-28T15:10:38.960310+0000","last_fullsized":"2024-03-28T15:10:38.960310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002201228,"stat_sum":{"num_bytes":6136325,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":89,"num_read_kb":4109,"num_write":317,"num_write_kb":10222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"37'225","reported_seq":285,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:30.368284+0000","last_change":"2024-03-28T15:01:31.095439+0000","last_active":"2024-03-28T15:10:30.368284+0000","last_peered":"2024-03-28T15:10:30.368284+0000","last_clean":"2024-03-28T15:10:30.368284+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:10:30.368284+0000","last_undegraded":"2024-03-28T15:10:30.368284+0000","last_fullsized":"2024-03-28T15:10:30.368284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:23.746086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":10,"num_read_kb":28674,"num_write":226,"num_write_kb":12578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"37'330","reported_seq":2634,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932774+0000","last_change":"2024-03-28T15:04:56.198312+0000","last_active":"2024-03-28T15:10:38.932774+0000","last_peered":"2024-03-28T15:10:38.932774+0000","last_clean":"2024-03-28T15:10:38.932774+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:10:38.932774+0000","last_undegraded":"2024-03-28T15:10:38.932774+0000","last_fullsized":"2024-03-28T15:10:38.932774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026072840000000001,"stat_sum":{"num_bytes":10172814,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":229,"num_read_kb":5026,"num_write":323,"num_write_kb":10109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'80","reported_seq":210,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.644362+0000","last_change":"2024-03-28T15:01:31.093265+0000","last_active":"2024-03-28T15:10:29.644362+0000","last_peered":"2024-03-28T15:10:29.644362+0000","last_clean":"2024-03-28T15:10:29.644362+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:10:29.644362+0000","last_undegraded":"2024-03-28T15:10:29.644362+0000","last_fullsized":"2024-03-28T15:10:29.644362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534189,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":569,"num_write":78,"num_write_kb":592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"37'335","reported_seq":2427,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938025+0000","last_change":"2024-03-28T15:04:56.293657+0000","last_active":"2024-03-28T15:10:38.938025+0000","last_peered":"2024-03-28T15:10:38.938025+0000","last_clean":"2024-03-28T15:10:38.938025+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:10:38.938025+0000","last_undegraded":"2024-03-28T15:10:38.938025+0000","last_fullsized":"2024-03-28T15:10:38.938025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017724730000000001,"stat_sum":{"num_bytes":1382309,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":326,"num_write_kb":1524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"25'82","reported_seq":140,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.264455+0000","last_change":"2024-03-28T15:01:31.094750+0000","last_active":"2024-03-28T15:10:01.264455+0000","last_peered":"2024-03-28T15:10:01.264455+0000","last_clean":"2024-03-28T15:10:01.264455+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:10:01.264455+0000","last_undegraded":"2024-03-28T15:10:01.264455+0000","last_fullsized":"2024-03-28T15:10:01.264455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":80,"num_write_kb":8230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"37'356","reported_seq":2673,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.933205+0000","last_change":"2024-03-28T15:04:56.198147+0000","last_active":"2024-03-28T15:10:38.933205+0000","last_peered":"2024-03-28T15:10:38.933205+0000","last_clean":"2024-03-28T15:10:38.933205+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:10:38.933205+0000","last_undegraded":"2024-03-28T15:10:38.933205+0000","last_fullsized":"2024-03-28T15:10:38.933205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:59.918813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075601100000000001,"stat_sum":{"num_bytes":5557804,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":243,"num_read_kb":4111,"num_write":349,"num_write_kb":5610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"36'123","reported_seq":178,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:03.306471+0000","last_change":"2024-03-28T15:01:31.093706+0000","last_active":"2024-03-28T15:10:03.306471+0000","last_peered":"2024-03-28T15:10:03.306471+0000","last_clean":"2024-03-28T15:10:03.306471+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:10:03.306471+0000","last_undegraded":"2024-03-28T15:10:03.306471+0000","last_fullsized":"2024-03-28T15:10:03.306471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4096,"num_write":123,"num_write_kb":4157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"37'346","reported_seq":2940,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956753+0000","last_change":"2024-03-28T15:04:56.200588+0000","last_active":"2024-03-28T15:10:38.956753+0000","last_peered":"2024-03-28T15:10:38.956753+0000","last_clean":"2024-03-28T15:10:38.956753+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:10:38.956753+0000","last_undegraded":"2024-03-28T15:10:38.956753+0000","last_fullsized":"2024-03-28T15:10:38.956753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:08.686987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038365610000000001,"stat_sum":{"num_bytes":18167101,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":520,"num_read_kb":17729,"num_write":341,"num_write_kb":26114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"37'1696","reported_seq":1753,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:23.448944+0000","last_change":"2024-03-28T15:01:31.095126+0000","last_active":"2024-03-28T15:10:23.448944+0000","last_peered":"2024-03-28T15:10:23.448944+0000","last_clean":"2024-03-28T15:10:23.448944+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:10:23.448944+0000","last_undegraded":"2024-03-28T15:10:23.448944+0000","last_fullsized":"2024-03-28T15:10:23.448944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":1696,"log_dups_size":0,"ondisk_log_size":1696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:21.669535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":12343,"num_write":1697,"num_write_kb":9020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"37'339","reported_seq":2634,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937869+0000","last_change":"2024-03-28T15:04:56.293414+0000","last_active":"2024-03-28T15:10:38.937869+0000","last_peered":"2024-03-28T15:10:38.937869+0000","last_clean":"2024-03-28T15:10:38.937869+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:10:38.937869+0000","last_undegraded":"2024-03-28T15:10:38.937869+0000","last_fullsized":"2024-03-28T15:10:38.937869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017181130000000001,"stat_sum":{"num_bytes":6397362,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":220,"num_read_kb":4361,"num_write":330,"num_write_kb":6420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'146","reported_seq":215,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:48.787578+0000","last_change":"2024-03-28T15:10:48.787629+0000","last_active":"2024-03-28T15:10:48.787578+0000","last_peered":"2024-03-28T15:10:48.787578+0000","last_clean":"2024-03-28T15:10:48.787578+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:10:48.787578+0000","last_undegraded":"2024-03-28T15:10:48.787578+0000","last_fullsized":"2024-03-28T15:10:48.787578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'146","last_scrub_stamp":"2024-03-28T15:10:48.787475+0000","last_deep_scrub":"37'146","last_deep_scrub_stamp":"2024-03-28T15:10:48.787475+0000","last_clean_scrub_stamp":"2024-03-28T15:10:48.787475+0000","objects_scrubbed":3,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:23:34.664993+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":239,"num_write":363,"num_write_kb":1821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":42500,"num_omap_keys":85,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"37'324","reported_seq":3171,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.946332+0000","last_change":"2024-03-28T15:04:56.208109+0000","last_active":"2024-03-28T15:10:38.946332+0000","last_peered":"2024-03-28T15:10:38.946332+0000","last_clean":"2024-03-28T15:10:38.946332+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:10:38.946332+0000","last_undegraded":"2024-03-28T15:10:38.946332+0000","last_fullsized":"2024-03-28T15:10:38.946332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002114524,"stat_sum":{"num_bytes":15625536,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":772,"num_read_kb":10204,"num_write":317,"num_write_kb":19524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"36'175","reported_seq":235,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:14.520739+0000","last_change":"2024-03-28T15:01:31.088204+0000","last_active":"2024-03-28T15:10:14.520739+0000","last_peered":"2024-03-28T15:10:14.520739+0000","last_clean":"2024-03-28T15:10:14.520739+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:10:14.520739+0000","last_undegraded":"2024-03-28T15:10:14.520739+0000","last_fullsized":"2024-03-28T15:10:14.520739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":4117,"num_write":175,"num_write_kb":4205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"37'353","reported_seq":3393,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.932206+0000","last_change":"2024-03-28T15:04:56.198120+0000","last_active":"2024-03-28T15:10:38.932206+0000","last_peered":"2024-03-28T15:10:38.932206+0000","last_clean":"2024-03-28T15:10:38.932206+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:10:38.932206+0000","last_undegraded":"2024-03-28T15:10:38.932206+0000","last_fullsized":"2024-03-28T15:10:38.932206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025149030000000002,"stat_sum":{"num_bytes":28522087,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":966,"num_read_kb":19238,"num_write":341,"num_write_kb":28027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'224","reported_seq":279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.644049+0000","last_change":"2024-03-28T15:01:31.095371+0000","last_active":"2024-03-28T15:10:29.644049+0000","last_peered":"2024-03-28T15:10:29.644049+0000","last_clean":"2024-03-28T15:10:29.644049+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:10:29.644049+0000","last_undegraded":"2024-03-28T15:10:29.644049+0000","last_fullsized":"2024-03-28T15:10:29.644049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:29.374853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":230,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"37'351","reported_seq":2841,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:33.541714+0000","last_change":"2024-03-28T15:04:56.196825+0000","last_active":"2024-03-28T15:10:33.541714+0000","last_peered":"2024-03-28T15:10:33.541714+0000","last_clean":"2024-03-28T15:10:33.541714+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:10:33.541714+0000","last_undegraded":"2024-03-28T15:10:33.541714+0000","last_fullsized":"2024-03-28T15:10:33.541714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:43:14.280586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083411100000000001,"stat_sum":{"num_bytes":27575979,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":416,"num_read_kb":10935,"num_write":349,"num_write_kb":27109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"36'18","reported_seq":72,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:14.524734+0000","last_change":"2024-03-28T15:01:31.094542+0000","last_active":"2024-03-28T15:10:14.524734+0000","last_peered":"2024-03-28T15:10:14.524734+0000","last_clean":"2024-03-28T15:10:14.524734+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:10:14.524734+0000","last_undegraded":"2024-03-28T15:10:14.524734+0000","last_fullsized":"2024-03-28T15:10:14.524734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":18,"num_write_kb":4204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"37'362","reported_seq":3012,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.938926+0000","last_change":"2024-03-28T15:04:56.208858+0000","last_active":"2024-03-28T15:10:38.938926+0000","last_peered":"2024-03-28T15:10:38.938926+0000","last_clean":"2024-03-28T15:10:38.938926+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:10:38.938926+0000","last_undegraded":"2024-03-28T15:10:38.938926+0000","last_fullsized":"2024-03-28T15:10:38.938926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015964149999999999,"stat_sum":{"num_bytes":23553308,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":575,"num_read_kb":17168,"num_write":351,"num_write_kb":27278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"37'29","reported_seq":85,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:32.561146+0000","last_change":"2024-03-28T15:01:31.094440+0000","last_active":"2024-03-28T15:10:32.561146+0000","last_peered":"2024-03-28T15:10:32.561146+0000","last_clean":"2024-03-28T15:10:32.561146+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:10:32.561146+0000","last_undegraded":"2024-03-28T15:10:32.561146+0000","last_fullsized":"2024-03-28T15:10:32.561146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":31,"num_write_kb":4146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"37'329","reported_seq":2883,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.937502+0000","last_change":"2024-03-28T15:04:56.208307+0000","last_active":"2024-03-28T15:10:38.937502+0000","last_peered":"2024-03-28T15:10:38.937502+0000","last_clean":"2024-03-28T15:10:38.937502+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:10:38.937502+0000","last_undegraded":"2024-03-28T15:10:38.937502+0000","last_fullsized":"2024-03-28T15:10:38.937502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022492050000000002,"stat_sum":{"num_bytes":15497791,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":479,"num_read_kb":3997,"num_write":321,"num_write_kb":15303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"37'250","reported_seq":315,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:49.412627+0000","last_change":"2024-03-28T15:01:31.096352+0000","last_active":"2024-03-28T15:10:49.412627+0000","last_peered":"2024-03-28T15:10:49.412627+0000","last_clean":"2024-03-28T15:10:49.412627+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:10:49.412627+0000","last_undegraded":"2024-03-28T15:10:49.412627+0000","last_fullsized":"2024-03-28T15:10:49.412627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:17.995389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9989805,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":18,"num_read_kb":28682,"num_write":245,"num_write_kb":9894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"37'352","reported_seq":2837,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.930151+0000","last_change":"2024-03-28T15:04:56.534400+0000","last_active":"2024-03-28T15:10:38.930151+0000","last_peered":"2024-03-28T15:10:38.930151+0000","last_clean":"2024-03-28T15:10:38.930151+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:10:38.930151+0000","last_undegraded":"2024-03-28T15:10:38.930151+0000","last_fullsized":"2024-03-28T15:10:38.930151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:43.521019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001719216,"stat_sum":{"num_bytes":11206715,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":411,"num_read_kb":8309,"num_write":347,"num_write_kb":11133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"37'327","reported_seq":2611,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.955085+0000","last_change":"2024-03-28T15:04:56.200097+0000","last_active":"2024-03-28T15:10:38.955085+0000","last_peered":"2024-03-28T15:10:38.955085+0000","last_clean":"2024-03-28T15:10:38.955085+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:10:38.955085+0000","last_undegraded":"2024-03-28T15:10:38.955085+0000","last_fullsized":"2024-03-28T15:10:38.955085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:17:15.635128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036538370000000001,"stat_sum":{"num_bytes":7862984,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":211,"num_read_kb":4119,"num_write":311,"num_write_kb":11936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"26'24","reported_seq":85,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:01.901412+0000","last_change":"2024-03-28T15:01:31.094281+0000","last_active":"2024-03-28T15:10:01.901412+0000","last_peered":"2024-03-28T15:10:01.901412+0000","last_clean":"2024-03-28T15:10:01.901412+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:10:01.901412+0000","last_undegraded":"2024-03-28T15:10:01.901412+0000","last_fullsized":"2024-03-28T15:10:01.901412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:19.742451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":24,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"37'344","reported_seq":3680,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.956194+0000","last_change":"2024-03-28T15:04:56.200342+0000","last_active":"2024-03-28T15:10:38.956194+0000","last_peered":"2024-03-28T15:10:38.956194+0000","last_clean":"2024-03-28T15:10:38.956194+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:10:38.956194+0000","last_undegraded":"2024-03-28T15:10:38.956194+0000","last_fullsized":"2024-03-28T15:10:38.956194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:51:56.779182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037913880000000001,"stat_sum":{"num_bytes":22932535,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":1263,"num_read_kb":23613,"num_write":335,"num_write_kb":26620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"37'170","reported_seq":237,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:29.643756+0000","last_change":"2024-03-28T15:01:31.093773+0000","last_active":"2024-03-28T15:10:29.643756+0000","last_peered":"2024-03-28T15:10:29.643756+0000","last_clean":"2024-03-28T15:10:29.643756+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:10:29.643756+0000","last_undegraded":"2024-03-28T15:10:29.643756+0000","last_fullsized":"2024-03-28T15:10:29.643756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:30.069469+0000","last_clean_scrub_stamp":"2024-03-28T15:01:30.069469+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:54.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":36885,"num_write":163,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"37'319","reported_seq":2905,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.943413+0000","last_change":"2024-03-28T15:04:56.197860+0000","last_active":"2024-03-28T15:10:38.943413+0000","last_peered":"2024-03-28T15:10:38.943413+0000","last_clean":"2024-03-28T15:10:38.943413+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:10:38.943413+0000","last_undegraded":"2024-03-28T15:10:38.943413+0000","last_fullsized":"2024-03-28T15:10:38.943413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023175320000000002,"stat_sum":{"num_bytes":14729150,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":511,"num_read_kb":13471,"num_write":311,"num_write_kb":14544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"37'1100","reported_seq":1164,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:49.159764+0000","last_change":"2024-03-28T15:10:49.159823+0000","last_active":"2024-03-28T15:10:49.159764+0000","last_peered":"2024-03-28T15:10:49.159764+0000","last_clean":"2024-03-28T15:10:49.159764+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:10:49.159764+0000","last_undegraded":"2024-03-28T15:10:49.159764+0000","last_fullsized":"2024-03-28T15:10:49.159764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'1100","last_scrub_stamp":"2024-03-28T15:10:49.159683+0000","last_deep_scrub":"37'1100","last_deep_scrub_stamp":"2024-03-28T15:10:49.159683+0000","last_clean_scrub_stamp":"2024-03-28T15:10:49.159683+0000","objects_scrubbed":1,"log_size":1100,"log_dups_size":0,"ondisk_log_size":1100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:58:02.691885+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":1093,"num_write_kb":5019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"37'346","reported_seq":2921,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:38.936302+0000","last_change":"2024-03-28T15:04:56.207406+0000","last_active":"2024-03-28T15:10:38.936302+0000","last_peered":"2024-03-28T15:10:38.936302+0000","last_clean":"2024-03-28T15:10:38.936302+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:10:38.936302+0000","last_undegraded":"2024-03-28T15:10:38.936302+0000","last_fullsized":"2024-03-28T15:10:38.936302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:01:31.071160+0000","last_clean_scrub_stamp":"2024-03-28T15:01:31.071160+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015914119999999999,"stat_sum":{"num_bytes":14776390,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":500,"num_read_kb":7280,"num_write":341,"num_write_kb":14609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"37'110","reported_seq":187,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:47.843363+0000","last_change":"2024-03-28T15:10:47.843457+0000","last_active":"2024-03-28T15:10:47.843363+0000","last_peered":"2024-03-28T15:10:47.843363+0000","last_clean":"2024-03-28T15:10:47.843363+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:10:47.843363+0000","last_undegraded":"2024-03-28T15:10:47.843363+0000","last_fullsized":"2024-03-28T15:10:47.843363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'110","last_scrub_stamp":"2024-03-28T15:10:47.843257+0000","last_deep_scrub":"37'110","last_deep_scrub_stamp":"2024-03-28T15:10:47.843257+0000","last_clean_scrub_stamp":"2024-03-28T15:10:47.843257+0000","objects_scrubbed":6,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:43.009374+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":20500,"num_write":104,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17374,"num_omap_keys":33,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":940897971,"num_objects":10589,"num_object_clones":0,"num_object_copies":21178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10589,"num_whiteouts":0,"num_read":30680,"num_read_kb":666406,"num_write":21173,"num_write_kb":1063512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1935458304,"data_stored":1880255846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21704,"ondisk_log_size":21704,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":182491928,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":492,"num_read_kb":424818,"num_write":6953,"num_write_kb":184815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":94136,"num_omap_keys":184,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":365043712,"data_stored":364983856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6766,"ondisk_log_size":6766,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574966,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425540,"kb_used_data":336124,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":93946300,"statfs":{"total":96636764160,"available":96201011200,"internally_reserved":0,"allocated":344190976,"data_stored":335783850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":91552202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574967,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294192,"kb_used_data":240808,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94077648,"statfs":{"total":96636764160,"available":96335511552,"internally_reserved":0,"allocated":246587392,"data_stored":241270326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574967,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354032,"kb_used_data":308968,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94017808,"statfs":{"total":96636764160,"available":96274235392,"internally_reserved":0,"allocated":316383232,"data_stored":308090025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574967,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":387812,"kb_used_data":338588,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93984028,"statfs":{"total":96636764160,"available":96239644672,"internally_reserved":0,"allocated":346714112,"data_stored":338268053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574966,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405916,"kb_used_data":360852,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93965924,"statfs":{"total":96636764160,"available":96221106176,"internally_reserved":0,"allocated":369512448,"data_stored":360982077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181040,"kb_used_data":149864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94190800,"statfs":{"total":96636764160,"available":96451379200,"internally_reserved":0,"allocated":153460736,"data_stored":148434971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267092,"kb_used_data":217868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94104748,"statfs":{"total":96636764160,"available":96363261952,"internally_reserved":0,"allocated":223096832,"data_stored":217893056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574966,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368928,"kb_used_data":297560,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94002912,"statfs":{"total":96636764160,"available":96258981888,"internally_reserved":0,"allocated":304701440,"data_stored":296790976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50331808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21508096,"data_stored":21505709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":34099200,"data_stored":34089293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51945472,"data_stored":51932971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50332320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":72908800,"data_stored":72904947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261738496,"data_stored":254075801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172343296,"data_stored":167380119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131538944,"data_stored":126748133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334999552,"data_stored":326711655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":294531072,"data_stored":286332021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":274022400,"data_stored":265965766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":195833856,"data_stored":190756877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":270450688,"data_stored":262285474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:10:51.375 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.375 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.376 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.376 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.376 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.376 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.376 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.376 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.377 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.377 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.377 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.377 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.377 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.377 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.378 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.378 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.378 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.378 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.378 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.378 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.378 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.379 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.379 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.379 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.379 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.379 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.379 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.380 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.380 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.380 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.380 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.380 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.380 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.381 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.381 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.381 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.381 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.381 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.381 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.381 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.382 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.382 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.382 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.382 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.382 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.383 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.383 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.383 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.383 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.383 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.383 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.383 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.384 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.384 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.384 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.384 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.384 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.384 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.385 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.385 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.385 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.385 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.385 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-28T15:01:19.548090+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=19, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.385 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.386 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.386 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.386 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.386 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.386 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.386 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.387 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.387 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.387 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.387 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.387 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.387 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.388 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.388 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.388 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.388 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.388 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.388 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.388 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.389 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.389 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.389 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.389 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.389 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.389 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.390 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-03-28T15:01:30.069469+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=30, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.390 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.391 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-28T15:01:31.071160+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=1, tm_sec=31, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=15, tm_min=10, tm_sec=46, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T15:10:51.391 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-28T15:10:57.784 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:10:57.786 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:11:11.393 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:11:11.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.505+0000 7f984740e640 1 Processor -- start 2024-03-28T15:11:11.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.505+0000 7f984740e640 1 -- start start 2024-03-28T15:11:11.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f984740e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9840151990 0x7f98400c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f984740e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98400c2d00 0x7f98400c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f984740e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f984006caa0 con 0x7f98400b1e30 2024-03-28T15:11:11.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f984740e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f98400c1da0 con 0x7f9840151990 2024-03-28T15:11:11.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f984740e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f984006cd80 con 0x7f98400c2d00 2024-03-28T15:11:11.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f98451aa640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9840151990 0x7f98400c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f98449a9640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f98400b1e30 0x7f98400712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:55538/0 (socket says 172.21.15.186:55538) 2024-03-28T15:11:11.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f98459ab640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98400c2d00 0x7f98400c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f98451aa640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9840151990 0x7f98400c2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54110/0 (socket says 172.21.15.186:54110) 2024-03-28T15:11:11.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f98449a9640 1 -- 172.21.15.186:0/2456088670 learned_addr learned my addr 172.21.15.186:0/2456088670 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:11:11.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f9837fff640 1 -- 172.21.15.186:0/2456088670 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129011858 0 0) 0x7f984006caa0 con 0x7f98400b1e30 2024-03-28T15:11:11.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f9837fff640 1 -- 172.21.15.186:0/2456088670 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f982c003520 con 0x7f98400b1e30 2024-03-28T15:11:11.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f98451aa640 1 -- 172.21.15.186:0/2456088670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98400c2d00 msgr2=0x7f98400c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:11:11.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f98451aa640 1 --2- 172.21.15.186:0/2456088670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98400c2d00 0x7f98400c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f98451aa640 1 -- 172.21.15.186:0/2456088670 >> v1:172.21.15.191:6789/0 conn(0x7f98400b1e30 legacy=0x7f98400712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:11.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f98451aa640 1 -- 172.21.15.186:0/2456088670 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98400c1340 con 0x7f9840151990 2024-03-28T15:11:11.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.506+0000 7f98459ab640 1 --2- 172.21.15.186:0/2456088670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98400c2d00 0x7f98400c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:11:11.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.507+0000 7f98451aa640 1 --2- 172.21.15.186:0/2456088670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9840151990 0x7f98400c2230 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9838009d90 tx=0x7f983802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf593d015a3ed45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:11.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.507+0000 7f9837fff640 1 -- 172.21.15.186:0/2456088670 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838039070 con 0x7f9840151990 2024-03-28T15:11:11.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.507+0000 7f9837fff640 1 -- 172.21.15.186:0/2456088670 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f983802cce0 con 0x7f9840151990 2024-03-28T15:11:11.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.507+0000 7f9837fff640 1 -- 172.21.15.186:0/2456088670 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838034660 con 0x7f9840151990 2024-03-28T15:11:11.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.507+0000 7f984740e640 1 -- 172.21.15.186:0/2456088670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9840151990 msgr2=0x7f98400c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:11.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.508+0000 7f984740e640 1 --2- 172.21.15.186:0/2456088670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9840151990 0x7f98400c2230 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9838009d90 tx=0x7f983802bf30 comp rx=0 tx=0).stop 2024-03-28T15:11:11.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.508+0000 7f984740e640 1 -- 172.21.15.186:0/2456088670 shutdown_connections 2024-03-28T15:11:11.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.508+0000 7f984740e640 1 --2- 172.21.15.186:0/2456088670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98400c2d00 0x7f98400c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.508+0000 7f984740e640 1 --2- 172.21.15.186:0/2456088670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9840151990 0x7f98400c2230 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.508+0000 7f984740e640 1 -- 172.21.15.186:0/2456088670 >> 172.21.15.186:0/2456088670 conn(0x7f98400c8a20 msgr2=0x7f98400c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:11.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.508+0000 7f984740e640 1 -- 172.21.15.186:0/2456088670 shutdown_connections 2024-03-28T15:11:11.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.508+0000 7f984740e640 1 -- 172.21.15.186:0/2456088670 wait complete. 2024-03-28T15:11:11.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.509+0000 7f984740e640 1 Processor -- start 2024-03-28T15:11:11.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.509+0000 7f984740e640 1 -- start start 2024-03-28T15:11:11.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.509+0000 7f984740e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400b1e30 0x7f984014db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.509+0000 7f984740e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98400c2d00 0x7f98400ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.509+0000 7f984740e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9840151990 0x7f9840141970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.509+0000 7f984740e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9840073dd0 con 0x7f98400b1e30 2024-03-28T15:11:11.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.509+0000 7f984740e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f98400c1900 con 0x7f9840151990 2024-03-28T15:11:11.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.509+0000 7f984740e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f9840073f50 con 0x7f98400c2d00 2024-03-28T15:11:11.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f98451aa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400b1e30 0x7f984014db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f98451aa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400b1e30 0x7f984014db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54124/0 (socket says 172.21.15.186:54124) 2024-03-28T15:11:11.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f98451aa640 1 -- 172.21.15.186:0/2746346868 learned_addr learned my addr 172.21.15.186:0/2746346868 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:11:11.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f98449a9640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98400c2d00 0x7f98400ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f98459ab640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9840151990 0x7f9840141970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f98451aa640 1 -- 172.21.15.186:0/2746346868 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98400c2d00 msgr2=0x7f98400ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:11.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f98451aa640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98400c2d00 0x7f98400ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f98451aa640 1 -- 172.21.15.186:0/2746346868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9840151990 msgr2=0x7f9840141970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:11.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f98451aa640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9840151990 0x7f9840141970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f98451aa640 1 -- 172.21.15.186:0/2746346868 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98401f1770 con 0x7f98400b1e30 2024-03-28T15:11:11.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f98451aa640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400b1e30 0x7f984014db10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9838009d90 tx=0x7f9838004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:11.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f9835ffb640 1 -- 172.21.15.186:0/2746346868 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838039050 con 0x7f98400b1e30 2024-03-28T15:11:11.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f9835ffb640 1 -- 172.21.15.186:0/2746346868 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9838045070 con 0x7f98400b1e30 2024-03-28T15:11:11.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f984740e640 1 -- 172.21.15.186:0/2746346868 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98400cb910 con 0x7f98400b1e30 2024-03-28T15:11:11.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.510+0000 7f9835ffb640 1 -- 172.21.15.186:0/2746346868 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838009450 con 0x7f98400b1e30 2024-03-28T15:11:11.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.511+0000 7f984740e640 1 -- 172.21.15.186:0/2746346868 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98401f1430 con 0x7f98400b1e30 2024-03-28T15:11:11.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.512+0000 7f984740e640 1 -- 172.21.15.186:0/2746346868 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9840069500 con 0x7f98400b1e30 2024-03-28T15:11:11.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.512+0000 7f9835ffb640 1 -- 172.21.15.186:0/2746346868 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f983803e850 con 0x7f98400b1e30 2024-03-28T15:11:11.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.512+0000 7f9835ffb640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f982406c530 0x7f982406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.513+0000 7f9835ffb640 1 -- 172.21.15.186:0/2746346868 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f98380ae6b0 con 0x7f98400b1e30 2024-03-28T15:11:11.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.513+0000 7f98449a9640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f982406c530 0x7f982406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.513+0000 7f98449a9640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f982406c530 0x7f982406e9f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f984015baf0 tx=0x7f9830009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:11.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.517+0000 7f9835ffb640 1 -- 172.21.15.186:0/2746346868 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f983807cbd0 con 0x7f98400b1e30 2024-03-28T15:11:11.691 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.690+0000 7f984740e640 1 -- 172.21.15.186:0/2746346868 --> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f98400b1a00 con 0x7f982406c530 2024-03-28T15:11:11.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.696+0000 7f9835ffb640 1 -- 172.21.15.186:0/2746346868 <== mgr.4101 v2:172.21.15.186:6832/2995751804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257117 (secure 0 0 0) 0x7f98400b1a00 con 0x7f982406c530 2024-03-28T15:11:11.700 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:11:11.701 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-03-28T15:11:11.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 -- 172.21.15.186:0/2746346868 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f982406c530 msgr2=0x7f982406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:11.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f982406c530 0x7f982406e9f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f984015baf0 tx=0x7f9830009040 comp rx=0 tx=0).stop 2024-03-28T15:11:11.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 -- 172.21.15.186:0/2746346868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400b1e30 msgr2=0x7f984014db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:11.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400b1e30 0x7f984014db10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9838009d90 tx=0x7f9838004a40 comp rx=0 tx=0).stop 2024-03-28T15:11:11.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 -- 172.21.15.186:0/2746346868 shutdown_connections 2024-03-28T15:11:11.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f982406c530 0x7f982406e9f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9840151990 0x7f9840141970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98400c2d00 0x7f98400ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 --2- 172.21.15.186:0/2746346868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98400b1e30 0x7f984014db10 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 -- 172.21.15.186:0/2746346868 >> 172.21.15.186:0/2746346868 conn(0x7f98400c8a20 msgr2=0x7f9840071560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:11.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 -- 172.21.15.186:0/2746346868 shutdown_connections 2024-03-28T15:11:11.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.699+0000 7f984740e640 1 -- 172.21.15.186:0/2746346868 wait complete. 2024-03-28T15:11:11.711 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-03-28T15:11:10.756036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1124159776,"num_objects":10721,"num_object_clones":0,"num_object_copies":21442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10721,"num_whiteouts":0,"num_read":31288,"num_read_kb":1091324,"num_write":28306,"num_write_kb":1250929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":591173,"num_omap_keys":1136,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38144,"ondisk_log_size":38144,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2803112,"kb_used_data":2251240,"kb_used_omap":12,"kb_used_meta":551795,"kb_avail":752171608,"statfs":{"total":773094113280,"available":770223726592,"internally_reserved":0,"allocated":2305269760,"data_stored":2248132948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":565038641},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":179509,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":155599,"num_omap_keys":301,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002778"},"pg_stats":[{"pgid":"3.3f","version":"37'504","reported_seq":3432,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:07.845050+0000","last_change":"2024-03-28T15:11:07.845109+0000","last_active":"2024-03-28T15:11:07.845050+0000","last_peered":"2024-03-28T15:11:07.845050+0000","last_clean":"2024-03-28T15:11:07.845050+0000","last_became_active":"2024-03-28T15:01:32.120971+0000","last_became_peered":"2024-03-28T15:01:32.120971+0000","last_unstale":"2024-03-28T15:11:07.845050+0000","last_undegraded":"2024-03-28T15:11:07.845050+0000","last_fullsized":"2024-03-28T15:11:07.845050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'504","last_scrub_stamp":"2024-03-28T15:11:07.844963+0000","last_deep_scrub":"37'504","last_deep_scrub_stamp":"2024-03-28T15:11:07.844963+0000","last_clean_scrub_stamp":"2024-03-28T15:11:07.844963+0000","objects_scrubbed":169,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:51.400482+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0.0024885649999999999,"stat_sum":{"num_bytes":26509954,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":839,"num_read_kb":12103,"num_write":327,"num_write_kb":26054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"37'463","reported_seq":2568,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:59.843333+0000","last_change":"2024-03-28T15:10:59.843370+0000","last_active":"2024-03-28T15:10:59.843333+0000","last_peered":"2024-03-28T15:10:59.843333+0000","last_clean":"2024-03-28T15:10:59.843333+0000","last_became_active":"2024-03-28T15:01:32.115540+0000","last_became_peered":"2024-03-28T15:01:32.115540+0000","last_unstale":"2024-03-28T15:10:59.843333+0000","last_undegraded":"2024-03-28T15:10:59.843333+0000","last_fullsized":"2024-03-28T15:10:59.843333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'463","last_scrub_stamp":"2024-03-28T15:10:59.843263+0000","last_deep_scrub":"37'463","last_deep_scrub_stamp":"2024-03-28T15:10:59.843263+0000","last_clean_scrub_stamp":"2024-03-28T15:10:59.843263+0000","objects_scrubbed":170,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:55:11.362432+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0.0019334860000000001,"stat_sum":{"num_bytes":1345083,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":314,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"37'466","reported_seq":3240,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:54.914890+0000","last_change":"2024-03-28T15:10:54.914917+0000","last_active":"2024-03-28T15:10:54.914890+0000","last_peered":"2024-03-28T15:10:54.914890+0000","last_clean":"2024-03-28T15:10:54.914890+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:10:54.914890+0000","last_undegraded":"2024-03-28T15:10:54.914890+0000","last_fullsized":"2024-03-28T15:10:54.914890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'466","last_scrub_stamp":"2024-03-28T15:10:54.914823+0000","last_deep_scrub":"37'466","last_deep_scrub_stamp":"2024-03-28T15:10:54.914823+0000","last_clean_scrub_stamp":"2024-03-28T15:10:54.914823+0000","objects_scrubbed":153,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:11:25.363829+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0.0025151280000000002,"stat_sum":{"num_bytes":12619399,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":685,"num_read_kb":17683,"num_write":321,"num_write_kb":20675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"37'461","reported_seq":3239,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:56.091938+0000","last_change":"2024-03-28T15:10:56.091976+0000","last_active":"2024-03-28T15:10:56.091938+0000","last_peered":"2024-03-28T15:10:56.091938+0000","last_clean":"2024-03-28T15:10:56.091938+0000","last_became_active":"2024-03-28T15:02:12.485356+0000","last_became_peered":"2024-03-28T15:02:12.485356+0000","last_unstale":"2024-03-28T15:10:56.091938+0000","last_undegraded":"2024-03-28T15:10:56.091938+0000","last_fullsized":"2024-03-28T15:10:56.091938+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'461","last_scrub_stamp":"2024-03-28T15:10:56.091847+0000","last_deep_scrub":"37'461","last_deep_scrub_stamp":"2024-03-28T15:10:56.091847+0000","last_clean_scrub_stamp":"2024-03-28T15:10:56.091847+0000","objects_scrubbed":171,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:10.981453+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0.0014203849999999999,"stat_sum":{"num_bytes":19355212,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":695,"num_read_kb":14278,"num_write":306,"num_write_kb":19012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"37'506","reported_seq":2709,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:51.894220+0000","last_change":"2024-03-28T15:10:51.894271+0000","last_active":"2024-03-28T15:10:51.894220+0000","last_peered":"2024-03-28T15:10:51.894220+0000","last_clean":"2024-03-28T15:10:51.894220+0000","last_became_active":"2024-03-28T15:01:32.112506+0000","last_became_peered":"2024-03-28T15:01:32.112506+0000","last_unstale":"2024-03-28T15:10:51.894220+0000","last_undegraded":"2024-03-28T15:10:51.894220+0000","last_fullsized":"2024-03-28T15:10:51.894220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'506","last_scrub_stamp":"2024-03-28T15:10:51.894130+0000","last_deep_scrub":"37'506","last_deep_scrub_stamp":"2024-03-28T15:10:51.894130+0000","last_clean_scrub_stamp":"2024-03-28T15:10:51.894130+0000","objects_scrubbed":163,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:21.466770+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0.002796064,"stat_sum":{"num_bytes":10679980,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":115,"num_read_kb":4114,"num_write":361,"num_write_kb":10554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"37'452","reported_seq":2778,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:59.190862+0000","last_change":"2024-03-28T15:10:59.190914+0000","last_active":"2024-03-28T15:10:59.190862+0000","last_peered":"2024-03-28T15:10:59.190862+0000","last_clean":"2024-03-28T15:10:59.190862+0000","last_became_active":"2024-03-28T15:01:32.115905+0000","last_became_peered":"2024-03-28T15:01:32.115905+0000","last_unstale":"2024-03-28T15:10:59.190862+0000","last_undegraded":"2024-03-28T15:10:59.190862+0000","last_fullsized":"2024-03-28T15:10:59.190862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'452","last_scrub_stamp":"2024-03-28T15:10:59.190789+0000","last_deep_scrub":"37'452","last_deep_scrub_stamp":"2024-03-28T15:10:59.190789+0000","last_clean_scrub_stamp":"2024-03-28T15:10:59.190789+0000","objects_scrubbed":149,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:09:32.740066+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0.003798718,"stat_sum":{"num_bytes":10128543,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":239,"num_read_kb":4123,"num_write":321,"num_write_kb":10056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"37'505","reported_seq":3621,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:04.862739+0000","last_change":"2024-03-28T15:11:04.862787+0000","last_active":"2024-03-28T15:11:04.862739+0000","last_peered":"2024-03-28T15:11:04.862739+0000","last_clean":"2024-03-28T15:11:04.862739+0000","last_became_active":"2024-03-28T15:01:32.114955+0000","last_became_peered":"2024-03-28T15:01:32.114955+0000","last_unstale":"2024-03-28T15:11:04.862739+0000","last_undegraded":"2024-03-28T15:11:04.862739+0000","last_fullsized":"2024-03-28T15:11:04.862739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'505","last_scrub_stamp":"2024-03-28T15:11:04.862630+0000","last_deep_scrub":"37'505","last_deep_scrub_stamp":"2024-03-28T15:11:04.862630+0000","last_clean_scrub_stamp":"2024-03-28T15:11:04.862630+0000","objects_scrubbed":151,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:29.958605+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0.001484766,"stat_sum":{"num_bytes":23645872,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":1028,"num_read_kb":17521,"num_write":350,"num_write_kb":23210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"37'504","reported_seq":3130,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:58.373555+0000","last_change":"2024-03-28T15:10:58.373588+0000","last_active":"2024-03-28T15:10:58.373555+0000","last_peered":"2024-03-28T15:10:58.373555+0000","last_clean":"2024-03-28T15:10:58.373555+0000","last_became_active":"2024-03-28T15:01:32.117954+0000","last_became_peered":"2024-03-28T15:01:32.117954+0000","last_unstale":"2024-03-28T15:10:58.373555+0000","last_undegraded":"2024-03-28T15:10:58.373555+0000","last_fullsized":"2024-03-28T15:10:58.373555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'504","last_scrub_stamp":"2024-03-28T15:10:58.373487+0000","last_deep_scrub":"37'504","last_deep_scrub_stamp":"2024-03-28T15:10:58.373487+0000","last_clean_scrub_stamp":"2024-03-28T15:10:58.373487+0000","objects_scrubbed":196,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:55.492823+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0.0017148910000000001,"stat_sum":{"num_bytes":19312172,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":537,"num_read_kb":7179,"num_write":326,"num_write_kb":18943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"37'498","reported_seq":2895,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:02.875382+0000","last_change":"2024-03-28T15:11:02.875450+0000","last_active":"2024-03-28T15:11:02.875382+0000","last_peered":"2024-03-28T15:11:02.875382+0000","last_clean":"2024-03-28T15:11:02.875382+0000","last_became_active":"2024-03-28T15:01:32.112810+0000","last_became_peered":"2024-03-28T15:01:32.112810+0000","last_unstale":"2024-03-28T15:11:02.875382+0000","last_undegraded":"2024-03-28T15:11:02.875382+0000","last_fullsized":"2024-03-28T15:11:02.875382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'498","last_scrub_stamp":"2024-03-28T15:11:02.875295+0000","last_deep_scrub":"37'498","last_deep_scrub_stamp":"2024-03-28T15:11:02.875295+0000","last_clean_scrub_stamp":"2024-03-28T15:11:02.875295+0000","objects_scrubbed":171,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:41:07.102566+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0.0023275240000000001,"stat_sum":{"num_bytes":6398085,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":309,"num_read_kb":8863,"num_write":320,"num_write_kb":10489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"37'455","reported_seq":3352,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:58.826299+0000","last_change":"2024-03-28T15:10:58.826343+0000","last_active":"2024-03-28T15:10:58.826299+0000","last_peered":"2024-03-28T15:10:58.826299+0000","last_clean":"2024-03-28T15:10:58.826299+0000","last_became_active":"2024-03-28T15:01:32.124914+0000","last_became_peered":"2024-03-28T15:01:32.124914+0000","last_unstale":"2024-03-28T15:10:58.826299+0000","last_undegraded":"2024-03-28T15:10:58.826299+0000","last_fullsized":"2024-03-28T15:10:58.826299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'455","last_scrub_stamp":"2024-03-28T15:10:58.826229+0000","last_deep_scrub":"37'455","last_deep_scrub_stamp":"2024-03-28T15:10:58.826229+0000","last_clean_scrub_stamp":"2024-03-28T15:10:58.826229+0000","objects_scrubbed":145,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:37:59.631870+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0.0019348060000000001,"stat_sum":{"num_bytes":19157707,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":810,"num_read_kb":17508,"num_write":311,"num_write_kb":18862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"37'413","reported_seq":3135,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:59.848045+0000","last_change":"2024-03-28T15:10:59.848073+0000","last_active":"2024-03-28T15:10:59.848045+0000","last_peered":"2024-03-28T15:10:59.848045+0000","last_clean":"2024-03-28T15:10:59.848045+0000","last_became_active":"2024-03-28T15:01:32.125519+0000","last_became_peered":"2024-03-28T15:01:32.125519+0000","last_unstale":"2024-03-28T15:10:59.848045+0000","last_undegraded":"2024-03-28T15:10:59.848045+0000","last_fullsized":"2024-03-28T15:10:59.848045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'413","last_scrub_stamp":"2024-03-28T15:10:59.847969+0000","last_deep_scrub":"37'413","last_deep_scrub_stamp":"2024-03-28T15:10:59.847969+0000","last_clean_scrub_stamp":"2024-03-28T15:10:59.847969+0000","objects_scrubbed":129,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:30:36.639028+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0.0018723170000000001,"stat_sum":{"num_bytes":18249049,"num_objects":129,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":635,"num_read_kb":13018,"num_write":297,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"37'417","reported_seq":2739,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:55.205446+0000","last_change":"2024-03-28T15:10:55.205486+0000","last_active":"2024-03-28T15:10:55.205446+0000","last_peered":"2024-03-28T15:10:55.205446+0000","last_clean":"2024-03-28T15:10:55.205446+0000","last_became_active":"2024-03-28T15:01:32.120530+0000","last_became_peered":"2024-03-28T15:01:32.120530+0000","last_unstale":"2024-03-28T15:10:55.205446+0000","last_undegraded":"2024-03-28T15:10:55.205446+0000","last_fullsized":"2024-03-28T15:10:55.205446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'417","last_scrub_stamp":"2024-03-28T15:10:55.205347+0000","last_deep_scrub":"37'417","last_deep_scrub_stamp":"2024-03-28T15:10:55.205347+0000","last_clean_scrub_stamp":"2024-03-28T15:10:55.205347+0000","objects_scrubbed":150,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:49:33.103075+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0.0013925400000000001,"stat_sum":{"num_bytes":6129603,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":234,"num_read_kb":4757,"num_write":288,"num_write_kb":10236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"37'424","reported_seq":3000,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:00.330427+0000","last_change":"2024-03-28T15:11:00.330462+0000","last_active":"2024-03-28T15:11:00.330427+0000","last_peered":"2024-03-28T15:11:00.330427+0000","last_clean":"2024-03-28T15:11:00.330427+0000","last_became_active":"2024-03-28T15:01:32.119087+0000","last_became_peered":"2024-03-28T15:01:32.119087+0000","last_unstale":"2024-03-28T15:11:00.330427+0000","last_undegraded":"2024-03-28T15:11:00.330427+0000","last_fullsized":"2024-03-28T15:11:00.330427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'424","last_scrub_stamp":"2024-03-28T15:11:00.330357+0000","last_deep_scrub":"37'424","last_deep_scrub_stamp":"2024-03-28T15:11:00.330357+0000","last_clean_scrub_stamp":"2024-03-28T15:11:00.330357+0000","objects_scrubbed":142,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:24:22.985448+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0.0017572180000000001,"stat_sum":{"num_bytes":15029757,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":489,"num_read_kb":12548,"num_write":286,"num_write_kb":23016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"37'491","reported_seq":2779,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:50.018603+0000","last_change":"2024-03-28T15:10:50.018603+0000","last_active":"2024-03-28T15:10:50.018603+0000","last_peered":"2024-03-28T15:10:50.018603+0000","last_clean":"2024-03-28T15:10:50.018603+0000","last_became_active":"2024-03-28T15:01:32.114002+0000","last_became_peered":"2024-03-28T15:01:32.114002+0000","last_unstale":"2024-03-28T15:10:50.018603+0000","last_undegraded":"2024-03-28T15:10:50.018603+0000","last_fullsized":"2024-03-28T15:10:50.018603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'491","last_scrub_stamp":"2024-03-28T15:10:50.018477+0000","last_deep_scrub":"37'491","last_deep_scrub_stamp":"2024-03-28T15:10:50.018477+0000","last_clean_scrub_stamp":"2024-03-28T15:10:50.018477+0000","objects_scrubbed":164,"log_size":491,"log_dups_size":0,"ondisk_log_size":491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:55:01.537646+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0.002253971,"stat_sum":{"num_bytes":6664925,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":199,"num_read_kb":4930,"num_write":335,"num_write_kb":6638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"37'521","reported_seq":3709,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:58.922701+0000","last_change":"2024-03-28T15:10:58.922740+0000","last_active":"2024-03-28T15:10:58.922701+0000","last_peered":"2024-03-28T15:10:58.922701+0000","last_clean":"2024-03-28T15:10:58.922701+0000","last_became_active":"2024-03-28T15:01:32.113102+0000","last_became_peered":"2024-03-28T15:01:32.113102+0000","last_unstale":"2024-03-28T15:10:58.922701+0000","last_undegraded":"2024-03-28T15:10:58.922701+0000","last_fullsized":"2024-03-28T15:10:58.922701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'521","last_scrub_stamp":"2024-03-28T15:10:58.922635+0000","last_deep_scrub":"37'521","last_deep_scrub_stamp":"2024-03-28T15:10:58.922635+0000","last_clean_scrub_stamp":"2024-03-28T15:10:58.922635+0000","objects_scrubbed":191,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:11.289720+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0.002179673,"stat_sum":{"num_bytes":23044368,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":1098,"num_read_kb":19203,"num_write":350,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"37'502","reported_seq":3073,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:58.275491+0000","last_change":"2024-03-28T15:10:58.275524+0000","last_active":"2024-03-28T15:10:58.275491+0000","last_peered":"2024-03-28T15:10:58.275491+0000","last_clean":"2024-03-28T15:10:58.275491+0000","last_became_active":"2024-03-28T15:01:32.115557+0000","last_became_peered":"2024-03-28T15:01:32.115557+0000","last_unstale":"2024-03-28T15:10:58.275491+0000","last_undegraded":"2024-03-28T15:10:58.275491+0000","last_fullsized":"2024-03-28T15:10:58.275491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'502","last_scrub_stamp":"2024-03-28T15:10:58.275429+0000","last_deep_scrub":"37'502","last_deep_scrub_stamp":"2024-03-28T15:10:58.275429+0000","last_clean_scrub_stamp":"2024-03-28T15:10:58.275429+0000","objects_scrubbed":171,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:09:42.339735+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0.0039634529999999996,"stat_sum":{"num_bytes":16147473,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":482,"num_read_kb":8871,"num_write":348,"num_write_kb":19203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"37'499","reported_seq":2739,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:57.289498+0000","last_change":"2024-03-28T15:10:57.289534+0000","last_active":"2024-03-28T15:10:57.289498+0000","last_peered":"2024-03-28T15:10:57.289498+0000","last_clean":"2024-03-28T15:10:57.289498+0000","last_became_active":"2024-03-28T15:01:32.112722+0000","last_became_peered":"2024-03-28T15:01:32.112722+0000","last_unstale":"2024-03-28T15:10:57.289498+0000","last_undegraded":"2024-03-28T15:10:57.289498+0000","last_fullsized":"2024-03-28T15:10:57.289498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'499","last_scrub_stamp":"2024-03-28T15:10:57.289425+0000","last_deep_scrub":"37'499","last_deep_scrub_stamp":"2024-03-28T15:10:57.289425+0000","last_clean_scrub_stamp":"2024-03-28T15:10:57.289425+0000","objects_scrubbed":167,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:17:39.341099+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0.0027424319999999999,"stat_sum":{"num_bytes":10498072,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":151,"num_read_kb":9005,"num_write":340,"num_write_kb":14498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"37'439","reported_seq":2557,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:54.038836+0000","last_change":"2024-03-28T15:10:54.038836+0000","last_active":"2024-03-28T15:10:54.038836+0000","last_peered":"2024-03-28T15:10:54.038836+0000","last_clean":"2024-03-28T15:10:54.038836+0000","last_became_active":"2024-03-28T15:01:32.119218+0000","last_became_peered":"2024-03-28T15:01:32.119218+0000","last_unstale":"2024-03-28T15:10:54.038836+0000","last_undegraded":"2024-03-28T15:10:54.038836+0000","last_fullsized":"2024-03-28T15:10:54.038836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'439","last_scrub_stamp":"2024-03-28T15:10:54.038737+0000","last_deep_scrub":"37'439","last_deep_scrub_stamp":"2024-03-28T15:10:54.038737+0000","last_clean_scrub_stamp":"2024-03-28T15:10:54.038737+0000","objects_scrubbed":146,"log_size":439,"log_dups_size":0,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:36:21.571517+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0.001179399,"stat_sum":{"num_bytes":9508598,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":30,"num_read_kb":42,"num_write":300,"num_write_kb":9445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"37'469","reported_seq":2870,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:00.884108+0000","last_change":"2024-03-28T15:11:00.884154+0000","last_active":"2024-03-28T15:11:00.884108+0000","last_peered":"2024-03-28T15:11:00.884108+0000","last_clean":"2024-03-28T15:11:00.884108+0000","last_became_active":"2024-03-28T15:01:32.111643+0000","last_became_peered":"2024-03-28T15:01:32.111643+0000","last_unstale":"2024-03-28T15:11:00.884108+0000","last_undegraded":"2024-03-28T15:11:00.884108+0000","last_fullsized":"2024-03-28T15:11:00.884108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'469","last_scrub_stamp":"2024-03-28T15:11:00.884000+0000","last_deep_scrub":"37'469","last_deep_scrub_stamp":"2024-03-28T15:11:00.884000+0000","last_clean_scrub_stamp":"2024-03-28T15:11:00.884000+0000","objects_scrubbed":159,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:02:01.575028+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0.0030062439999999999,"stat_sum":{"num_bytes":14185939,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":312,"num_read_kb":12813,"num_write":317,"num_write_kb":18119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"37'495","reported_seq":3442,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:56.843889+0000","last_change":"2024-03-28T15:10:56.843943+0000","last_active":"2024-03-28T15:10:56.843889+0000","last_peered":"2024-03-28T15:10:56.843889+0000","last_clean":"2024-03-28T15:10:56.843889+0000","last_became_active":"2024-03-28T15:01:32.123952+0000","last_became_peered":"2024-03-28T15:01:32.123952+0000","last_unstale":"2024-03-28T15:10:56.843889+0000","last_undegraded":"2024-03-28T15:10:56.843889+0000","last_fullsized":"2024-03-28T15:10:56.843889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'495","last_scrub_stamp":"2024-03-28T15:10:56.843784+0000","last_deep_scrub":"37'495","last_deep_scrub_stamp":"2024-03-28T15:10:56.843784+0000","last_clean_scrub_stamp":"2024-03-28T15:10:56.843784+0000","objects_scrubbed":164,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:40.088728+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0.001798584,"stat_sum":{"num_bytes":30036364,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":858,"num_read_kb":15492,"num_write":348,"num_write_kb":37706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"37'497","reported_seq":2894,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:55.891495+0000","last_change":"2024-03-28T15:10:55.891546+0000","last_active":"2024-03-28T15:10:55.891495+0000","last_peered":"2024-03-28T15:10:55.891495+0000","last_clean":"2024-03-28T15:10:55.891495+0000","last_became_active":"2024-03-28T15:01:32.119830+0000","last_became_peered":"2024-03-28T15:01:32.119830+0000","last_unstale":"2024-03-28T15:10:55.891495+0000","last_undegraded":"2024-03-28T15:10:55.891495+0000","last_fullsized":"2024-03-28T15:10:55.891495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'497","last_scrub_stamp":"2024-03-28T15:10:55.891424+0000","last_deep_scrub":"37'497","last_deep_scrub_stamp":"2024-03-28T15:10:55.891424+0000","last_clean_scrub_stamp":"2024-03-28T15:10:55.891424+0000","objects_scrubbed":173,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:58:09.423625+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0.0025216930000000002,"stat_sum":{"num_bytes":10421413,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":308,"num_read_kb":8356,"num_write":331,"num_write_kb":10350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"37'472","reported_seq":2673,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:55.085596+0000","last_change":"2024-03-28T15:10:55.085622+0000","last_active":"2024-03-28T15:10:55.085596+0000","last_peered":"2024-03-28T15:10:55.085596+0000","last_clean":"2024-03-28T15:10:55.085596+0000","last_became_active":"2024-03-28T15:02:12.178196+0000","last_became_peered":"2024-03-28T15:02:12.178196+0000","last_unstale":"2024-03-28T15:10:55.085596+0000","last_undegraded":"2024-03-28T15:10:55.085596+0000","last_fullsized":"2024-03-28T15:10:55.085596+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'472","last_scrub_stamp":"2024-03-28T15:10:55.085519+0000","last_deep_scrub":"37'472","last_deep_scrub_stamp":"2024-03-28T15:10:55.085519+0000","last_clean_scrub_stamp":"2024-03-28T15:10:55.085519+0000","objects_scrubbed":155,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:22.696079+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0.001195878,"stat_sum":{"num_bytes":9431253,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":119,"num_read_kb":4113,"num_write":327,"num_write_kb":13479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"37'464","reported_seq":3146,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:53.301159+0000","last_change":"2024-03-28T15:10:53.301185+0000","last_active":"2024-03-28T15:10:53.301159+0000","last_peered":"2024-03-28T15:10:53.301159+0000","last_clean":"2024-03-28T15:10:53.301159+0000","last_became_active":"2024-03-28T15:01:32.113063+0000","last_became_peered":"2024-03-28T15:01:32.113063+0000","last_unstale":"2024-03-28T15:10:53.301159+0000","last_undegraded":"2024-03-28T15:10:53.301159+0000","last_fullsized":"2024-03-28T15:10:53.301159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'464","last_scrub_stamp":"2024-03-28T15:10:53.301079+0000","last_deep_scrub":"37'464","last_deep_scrub_stamp":"2024-03-28T15:10:53.301079+0000","last_clean_scrub_stamp":"2024-03-28T15:10:53.301079+0000","objects_scrubbed":152,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:55:33.755251+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0.0020004409999999999,"stat_sum":{"num_bytes":16319296,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":593,"num_read_kb":18222,"num_write":323,"num_write_kb":20203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"37'507","reported_seq":2966,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:55.321497+0000","last_change":"2024-03-28T15:10:55.321521+0000","last_active":"2024-03-28T15:10:55.321497+0000","last_peered":"2024-03-28T15:10:55.321497+0000","last_clean":"2024-03-28T15:10:55.321497+0000","last_became_active":"2024-03-28T15:01:32.112066+0000","last_became_peered":"2024-03-28T15:01:32.112066+0000","last_unstale":"2024-03-28T15:10:55.321497+0000","last_undegraded":"2024-03-28T15:10:55.321497+0000","last_fullsized":"2024-03-28T15:10:55.321497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'507","last_scrub_stamp":"2024-03-28T15:10:55.321434+0000","last_deep_scrub":"37'507","last_deep_scrub_stamp":"2024-03-28T15:10:55.321434+0000","last_clean_scrub_stamp":"2024-03-28T15:10:55.321434+0000","objects_scrubbed":177,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:38.170037+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0.0031981639999999999,"stat_sum":{"num_bytes":14893759,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":369,"num_read_kb":10097,"num_write":349,"num_write_kb":14727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"37'456","reported_seq":3412,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:02.392850+0000","last_change":"2024-03-28T15:11:02.392872+0000","last_active":"2024-03-28T15:11:02.392850+0000","last_peered":"2024-03-28T15:11:02.392850+0000","last_clean":"2024-03-28T15:11:02.392850+0000","last_became_active":"2024-03-28T15:01:32.122132+0000","last_became_peered":"2024-03-28T15:01:32.122132+0000","last_unstale":"2024-03-28T15:11:02.392850+0000","last_undegraded":"2024-03-28T15:11:02.392850+0000","last_fullsized":"2024-03-28T15:11:02.392850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'456","last_scrub_stamp":"2024-03-28T15:11:02.392768+0000","last_deep_scrub":"37'456","last_deep_scrub_stamp":"2024-03-28T15:11:02.392768+0000","last_clean_scrub_stamp":"2024-03-28T15:11:02.392768+0000","objects_scrubbed":150,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:31:31.706343+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0.001813119,"stat_sum":{"num_bytes":18699537,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":867,"num_read_kb":15099,"num_write":310,"num_write_kb":18421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"37'526","reported_seq":2782,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:55.771785+0000","last_change":"2024-03-28T15:10:55.771858+0000","last_active":"2024-03-28T15:10:55.771785+0000","last_peered":"2024-03-28T15:10:55.771785+0000","last_clean":"2024-03-28T15:10:55.771785+0000","last_became_active":"2024-03-28T15:01:32.123580+0000","last_became_peered":"2024-03-28T15:01:32.123580+0000","last_unstale":"2024-03-28T15:10:55.771785+0000","last_undegraded":"2024-03-28T15:10:55.771785+0000","last_fullsized":"2024-03-28T15:10:55.771785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'526","last_scrub_stamp":"2024-03-28T15:10:55.771700+0000","last_deep_scrub":"37'526","last_deep_scrub_stamp":"2024-03-28T15:10:55.771700+0000","last_clean_scrub_stamp":"2024-03-28T15:10:55.771700+0000","objects_scrubbed":174,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:17:23.607459+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0.0019272180000000001,"stat_sum":{"num_bytes":11206818,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":167,"num_read_kb":4115,"num_write":363,"num_write_kb":11124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"37'474","reported_seq":3093,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:55.947549+0000","last_change":"2024-03-28T15:10:55.947584+0000","last_active":"2024-03-28T15:10:55.947549+0000","last_peered":"2024-03-28T15:10:55.947549+0000","last_clean":"2024-03-28T15:10:55.947549+0000","last_became_active":"2024-03-28T15:01:32.110112+0000","last_became_peered":"2024-03-28T15:01:32.110112+0000","last_unstale":"2024-03-28T15:10:55.947549+0000","last_undegraded":"2024-03-28T15:10:55.947549+0000","last_fullsized":"2024-03-28T15:10:55.947549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'474","last_scrub_stamp":"2024-03-28T15:10:55.947481+0000","last_deep_scrub":"37'474","last_deep_scrub_stamp":"2024-03-28T15:10:55.947481+0000","last_clean_scrub_stamp":"2024-03-28T15:10:55.947481+0000","objects_scrubbed":180,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:54.228072+0000","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0.0019373070000000001,"stat_sum":{"num_bytes":22270707,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":529,"num_read_kb":12598,"num_write":314,"num_write_kb":21911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"37'448","reported_seq":3375,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:54.333037+0000","last_change":"2024-03-28T15:10:54.333068+0000","last_active":"2024-03-28T15:10:54.333037+0000","last_peered":"2024-03-28T15:10:54.333037+0000","last_clean":"2024-03-28T15:10:54.333037+0000","last_became_active":"2024-03-28T15:01:32.113119+0000","last_became_peered":"2024-03-28T15:01:32.113119+0000","last_unstale":"2024-03-28T15:10:54.333037+0000","last_undegraded":"2024-03-28T15:10:54.333037+0000","last_fullsized":"2024-03-28T15:10:54.333037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'448","last_scrub_stamp":"2024-03-28T15:10:54.332965+0000","last_deep_scrub":"37'448","last_deep_scrub_stamp":"2024-03-28T15:10:54.332965+0000","last_clean_scrub_stamp":"2024-03-28T15:10:54.332965+0000","objects_scrubbed":159,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:51:09.079377+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0.0020980170000000002,"stat_sum":{"num_bytes":17909578,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":838,"num_read_kb":18539,"num_write":311,"num_write_kb":20928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"37'500","reported_seq":3349,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:53.914523+0000","last_change":"2024-03-28T15:10:53.914555+0000","last_active":"2024-03-28T15:10:53.914523+0000","last_peered":"2024-03-28T15:10:53.914523+0000","last_clean":"2024-03-28T15:10:53.914523+0000","last_became_active":"2024-03-28T15:01:32.117274+0000","last_became_peered":"2024-03-28T15:01:32.117274+0000","last_unstale":"2024-03-28T15:10:53.914523+0000","last_undegraded":"2024-03-28T15:10:53.914523+0000","last_fullsized":"2024-03-28T15:10:53.914523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'500","last_scrub_stamp":"2024-03-28T15:10:53.914453+0000","last_deep_scrub":"37'500","last_deep_scrub_stamp":"2024-03-28T15:10:53.914453+0000","last_clean_scrub_stamp":"2024-03-28T15:10:53.914453+0000","objects_scrubbed":166,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:20.765113+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0.0026139499999999999,"stat_sum":{"num_bytes":26288347,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":760,"num_read_kb":8154,"num_write":350,"num_write_kb":25854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"37'508","reported_seq":3064,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:00.156664+0000","last_change":"2024-03-28T15:11:00.156696+0000","last_active":"2024-03-28T15:11:00.156664+0000","last_peered":"2024-03-28T15:11:00.156664+0000","last_clean":"2024-03-28T15:11:00.156664+0000","last_became_active":"2024-03-28T15:01:32.117119+0000","last_became_peered":"2024-03-28T15:01:32.117119+0000","last_unstale":"2024-03-28T15:11:00.156664+0000","last_undegraded":"2024-03-28T15:11:00.156664+0000","last_fullsized":"2024-03-28T15:11:00.156664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'508","last_scrub_stamp":"2024-03-28T15:11:00.156592+0000","last_deep_scrub":"37'508","last_deep_scrub_stamp":"2024-03-28T15:11:00.156592+0000","last_clean_scrub_stamp":"2024-03-28T15:11:00.156592+0000","objects_scrubbed":164,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:05:50.368770+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0.0036450829999999999,"stat_sum":{"num_bytes":10387544,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":467,"num_read_kb":8853,"num_write":352,"num_write_kb":10266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"37'527","reported_seq":3124,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:53.042333+0000","last_change":"2024-03-28T15:10:53.042368+0000","last_active":"2024-03-28T15:10:53.042333+0000","last_peered":"2024-03-28T15:10:53.042333+0000","last_clean":"2024-03-28T15:10:53.042333+0000","last_became_active":"2024-03-28T15:01:32.111061+0000","last_became_peered":"2024-03-28T15:01:32.111061+0000","last_unstale":"2024-03-28T15:10:53.042333+0000","last_undegraded":"2024-03-28T15:10:53.042333+0000","last_fullsized":"2024-03-28T15:10:53.042333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'527","last_scrub_stamp":"2024-03-28T15:10:53.042245+0000","last_deep_scrub":"37'527","last_deep_scrub_stamp":"2024-03-28T15:10:53.042245+0000","last_clean_scrub_stamp":"2024-03-28T15:10:53.042245+0000","objects_scrubbed":174,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:00.195181+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0.001020962,"stat_sum":{"num_bytes":12939697,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":508,"num_read_kb":8898,"num_write":372,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"37'547","reported_seq":2966,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:01.802390+0000","last_change":"2024-03-28T15:11:01.802438+0000","last_active":"2024-03-28T15:11:01.802390+0000","last_peered":"2024-03-28T15:11:01.802390+0000","last_clean":"2024-03-28T15:11:01.802390+0000","last_became_active":"2024-03-28T15:01:32.126500+0000","last_became_peered":"2024-03-28T15:01:32.126500+0000","last_unstale":"2024-03-28T15:11:01.802390+0000","last_undegraded":"2024-03-28T15:11:01.802390+0000","last_fullsized":"2024-03-28T15:11:01.802390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'547","last_scrub_stamp":"2024-03-28T15:11:01.802300+0000","last_deep_scrub":"37'547","last_deep_scrub_stamp":"2024-03-28T15:11:01.802300+0000","last_clean_scrub_stamp":"2024-03-28T15:11:01.802300+0000","objects_scrubbed":180,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:55:13.321469+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0.0019055229999999999,"stat_sum":{"num_bytes":11304826,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":329,"num_read_kb":5287,"num_write":374,"num_write_kb":11236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"37'464","reported_seq":3011,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:52.237681+0000","last_change":"2024-03-28T15:10:52.237710+0000","last_active":"2024-03-28T15:10:52.237681+0000","last_peered":"2024-03-28T15:10:52.237681+0000","last_clean":"2024-03-28T15:10:52.237681+0000","last_became_active":"2024-03-28T15:01:32.111408+0000","last_became_peered":"2024-03-28T15:01:32.111408+0000","last_unstale":"2024-03-28T15:10:52.237681+0000","last_undegraded":"2024-03-28T15:10:52.237681+0000","last_fullsized":"2024-03-28T15:10:52.237681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'464","last_scrub_stamp":"2024-03-28T15:10:52.237615+0000","last_deep_scrub":"37'464","last_deep_scrub_stamp":"2024-03-28T15:10:52.237615+0000","last_clean_scrub_stamp":"2024-03-28T15:10:52.237615+0000","objects_scrubbed":159,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:19:15.227823+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0.00078031800000000001,"stat_sum":{"num_bytes":14482293,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":459,"num_read_kb":10396,"num_write":315,"num_write_kb":14305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"37'21","reported_seq":86,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:50.177102+0000","last_change":"2024-03-28T15:10:50.177150+0000","last_active":"2024-03-28T15:10:50.177102+0000","last_peered":"2024-03-28T15:10:50.177102+0000","last_clean":"2024-03-28T15:10:50.177102+0000","last_became_active":"2024-03-28T15:01:31.097151+0000","last_became_peered":"2024-03-28T15:01:31.097151+0000","last_unstale":"2024-03-28T15:10:50.177102+0000","last_undegraded":"2024-03-28T15:10:50.177102+0000","last_fullsized":"2024-03-28T15:10:50.177102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'21","last_scrub_stamp":"2024-03-28T15:10:50.177016+0000","last_deep_scrub":"37'21","last_deep_scrub_stamp":"2024-03-28T15:10:50.177016+0000","last_clean_scrub_stamp":"2024-03-28T15:10:50.177016+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:04:31.544956+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":16384,"num_write":21,"num_write_kb":4230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8570,"num_omap_keys":16,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"37'453","reported_seq":2781,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:57.887431+0000","last_change":"2024-03-28T15:10:57.887459+0000","last_active":"2024-03-28T15:10:57.887431+0000","last_peered":"2024-03-28T15:10:57.887431+0000","last_clean":"2024-03-28T15:10:57.887431+0000","last_became_active":"2024-03-28T15:01:32.123278+0000","last_became_peered":"2024-03-28T15:01:32.123278+0000","last_unstale":"2024-03-28T15:10:57.887431+0000","last_undegraded":"2024-03-28T15:10:57.887431+0000","last_fullsized":"2024-03-28T15:10:57.887431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'453","last_scrub_stamp":"2024-03-28T15:10:57.887362+0000","last_deep_scrub":"37'453","last_deep_scrub_stamp":"2024-03-28T15:10:57.887362+0000","last_clean_scrub_stamp":"2024-03-28T15:10:57.887362+0000","objects_scrubbed":153,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:42.130201+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0.0023690579999999998,"stat_sum":{"num_bytes":11043752,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":240,"num_read_kb":4317,"num_write":317,"num_write_kb":10950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"37'651","reported_seq":724,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:52.677152+0000","last_change":"2024-03-28T15:10:52.677201+0000","last_active":"2024-03-28T15:10:52.677152+0000","last_peered":"2024-03-28T15:10:52.677152+0000","last_clean":"2024-03-28T15:10:52.677152+0000","last_became_active":"2024-03-28T15:01:31.088156+0000","last_became_peered":"2024-03-28T15:01:31.088156+0000","last_unstale":"2024-03-28T15:10:52.677152+0000","last_undegraded":"2024-03-28T15:10:52.677152+0000","last_fullsized":"2024-03-28T15:10:52.677152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'651","last_scrub_stamp":"2024-03-28T15:10:52.677064+0000","last_deep_scrub":"37'651","last_deep_scrub_stamp":"2024-03-28T15:10:52.677064+0000","last_clean_scrub_stamp":"2024-03-28T15:10:52.677064+0000","objects_scrubbed":4,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:18:32.910004+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":34526,"num_write":642,"num_write_kb":12757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11836,"num_omap_keys":22,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"37'65","reported_seq":128,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:51.166484+0000","last_change":"2024-03-28T15:10:51.166544+0000","last_active":"2024-03-28T15:10:51.166484+0000","last_peered":"2024-03-28T15:10:51.166484+0000","last_clean":"2024-03-28T15:10:51.166484+0000","last_became_active":"2024-03-28T15:01:31.097249+0000","last_became_peered":"2024-03-28T15:01:31.097249+0000","last_unstale":"2024-03-28T15:10:51.166484+0000","last_undegraded":"2024-03-28T15:10:51.166484+0000","last_fullsized":"2024-03-28T15:10:51.166484+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'65","last_scrub_stamp":"2024-03-28T15:10:51.166393+0000","last_deep_scrub":"37'65","last_deep_scrub_stamp":"2024-03-28T15:10:51.166393+0000","last_clean_scrub_stamp":"2024-03-28T15:10:51.166393+0000","objects_scrubbed":7,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:03:49.604016+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":62,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":57042,"num_omap_keys":107,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"37'560","reported_seq":3867,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:57.407400+0000","last_change":"2024-03-28T15:10:57.407432+0000","last_active":"2024-03-28T15:10:57.407400+0000","last_peered":"2024-03-28T15:10:57.407400+0000","last_clean":"2024-03-28T15:10:57.407400+0000","last_became_active":"2024-03-28T15:01:32.115385+0000","last_became_peered":"2024-03-28T15:01:32.115385+0000","last_unstale":"2024-03-28T15:10:57.407400+0000","last_undegraded":"2024-03-28T15:10:57.407400+0000","last_fullsized":"2024-03-28T15:10:57.407400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'560","last_scrub_stamp":"2024-03-28T15:10:57.407325+0000","last_deep_scrub":"37'560","last_deep_scrub_stamp":"2024-03-28T15:10:57.407325+0000","last_clean_scrub_stamp":"2024-03-28T15:10:57.407325+0000","objects_scrubbed":188,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:17:39.458998+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0.0017492600000000001,"stat_sum":{"num_bytes":22338843,"num_objects":188,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":1218,"num_read_kb":22454,"num_write":394,"num_write_kb":30214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"37'609","reported_seq":675,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:51.286098+0000","last_change":"2024-03-28T15:10:51.286128+0000","last_active":"2024-03-28T15:10:51.286098+0000","last_peered":"2024-03-28T15:10:51.286098+0000","last_clean":"2024-03-28T15:10:51.286098+0000","last_became_active":"2024-03-28T15:01:31.085966+0000","last_became_peered":"2024-03-28T15:01:31.085966+0000","last_unstale":"2024-03-28T15:10:51.286098+0000","last_undegraded":"2024-03-28T15:10:51.286098+0000","last_fullsized":"2024-03-28T15:10:51.286098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'609","last_scrub_stamp":"2024-03-28T15:10:51.286019+0000","last_deep_scrub":"37'609","last_deep_scrub_stamp":"2024-03-28T15:10:51.286019+0000","last_clean_scrub_stamp":"2024-03-28T15:10:51.286019+0000","objects_scrubbed":6,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:35.528858+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":603,"num_write_kb":12492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":23672,"num_omap_keys":44,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"37'485","reported_seq":3506,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:56.959869+0000","last_change":"2024-03-28T15:10:56.959904+0000","last_active":"2024-03-28T15:10:56.959869+0000","last_peered":"2024-03-28T15:10:56.959869+0000","last_clean":"2024-03-28T15:10:56.959869+0000","last_became_active":"2024-03-28T15:01:32.110161+0000","last_became_peered":"2024-03-28T15:01:32.110161+0000","last_unstale":"2024-03-28T15:10:56.959869+0000","last_undegraded":"2024-03-28T15:10:56.959869+0000","last_fullsized":"2024-03-28T15:10:56.959869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'485","last_scrub_stamp":"2024-03-28T15:10:56.959802+0000","last_deep_scrub":"37'485","last_deep_scrub_stamp":"2024-03-28T15:10:56.959802+0000","last_clean_scrub_stamp":"2024-03-28T15:10:56.959802+0000","objects_scrubbed":172,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:37:57.765443+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0.0022536280000000001,"stat_sum":{"num_bytes":19647424,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":933,"num_read_kb":22521,"num_write":330,"num_write_kb":23406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"37'5","reported_seq":65,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:54.657036+0000","last_change":"2024-03-28T15:10:54.657060+0000","last_active":"2024-03-28T15:10:54.657036+0000","last_peered":"2024-03-28T15:10:54.657036+0000","last_clean":"2024-03-28T15:10:54.657036+0000","last_became_active":"2024-03-28T15:01:31.091949+0000","last_became_peered":"2024-03-28T15:01:31.091949+0000","last_unstale":"2024-03-28T15:10:54.657036+0000","last_undegraded":"2024-03-28T15:10:54.657036+0000","last_fullsized":"2024-03-28T15:10:54.657036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'5","last_scrub_stamp":"2024-03-28T15:10:54.656961+0000","last_deep_scrub":"37'5","last_deep_scrub_stamp":"2024-03-28T15:10:54.656961+0000","last_clean_scrub_stamp":"2024-03-28T15:10:54.656961+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:18.511808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8918,"num_omap_keys":17,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"37'522","reported_seq":3399,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:02.168803+0000","last_change":"2024-03-28T15:11:02.168839+0000","last_active":"2024-03-28T15:11:02.168803+0000","last_peered":"2024-03-28T15:11:02.168803+0000","last_clean":"2024-03-28T15:11:02.168803+0000","last_became_active":"2024-03-28T15:01:32.121367+0000","last_became_peered":"2024-03-28T15:01:32.121367+0000","last_unstale":"2024-03-28T15:11:02.168803+0000","last_undegraded":"2024-03-28T15:11:02.168803+0000","last_fullsized":"2024-03-28T15:11:02.168803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'522","last_scrub_stamp":"2024-03-28T15:11:02.168733+0000","last_deep_scrub":"37'522","last_deep_scrub_stamp":"2024-03-28T15:11:02.168733+0000","last_clean_scrub_stamp":"2024-03-28T15:11:02.168733+0000","objects_scrubbed":166,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:52.310393+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0.0043389099999999996,"stat_sum":{"num_bytes":22872414,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":789,"num_read_kb":19564,"num_write":357,"num_write_kb":22493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"37'71","reported_seq":134,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:52.266086+0000","last_change":"2024-03-28T15:10:52.266129+0000","last_active":"2024-03-28T15:10:52.266086+0000","last_peered":"2024-03-28T15:10:52.266086+0000","last_clean":"2024-03-28T15:10:52.266086+0000","last_became_active":"2024-03-28T15:01:31.088589+0000","last_became_peered":"2024-03-28T15:01:31.088589+0000","last_unstale":"2024-03-28T15:10:52.266086+0000","last_undegraded":"2024-03-28T15:10:52.266086+0000","last_fullsized":"2024-03-28T15:10:52.266086+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'71","last_scrub_stamp":"2024-03-28T15:10:52.265978+0000","last_deep_scrub":"37'71","last_deep_scrub_stamp":"2024-03-28T15:10:52.265978+0000","last_clean_scrub_stamp":"2024-03-28T15:10:52.265978+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:28:31.914509+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":66,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"37'451","reported_seq":3112,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:51.937253+0000","last_change":"2024-03-28T15:10:51.937292+0000","last_active":"2024-03-28T15:10:51.937253+0000","last_peered":"2024-03-28T15:10:51.937253+0000","last_clean":"2024-03-28T15:10:51.937253+0000","last_became_active":"2024-03-28T15:01:32.116879+0000","last_became_peered":"2024-03-28T15:01:32.116879+0000","last_unstale":"2024-03-28T15:10:51.937253+0000","last_undegraded":"2024-03-28T15:10:51.937253+0000","last_fullsized":"2024-03-28T15:10:51.937253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'451","last_scrub_stamp":"2024-03-28T15:10:51.937181+0000","last_deep_scrub":"37'451","last_deep_scrub_stamp":"2024-03-28T15:10:51.937181+0000","last_clean_scrub_stamp":"2024-03-28T15:10:51.937181+0000","objects_scrubbed":152,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:17:35.336688+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0.0023945189999999999,"stat_sum":{"num_bytes":10186807,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":573,"num_read_kb":8919,"num_write":314,"num_write_kb":10118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"37'199","reported_seq":272,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:52.735802+0000","last_change":"2024-03-28T15:10:52.735837+0000","last_active":"2024-03-28T15:10:52.735802+0000","last_peered":"2024-03-28T15:10:52.735802+0000","last_clean":"2024-03-28T15:10:52.735802+0000","last_became_active":"2024-03-28T15:01:31.095642+0000","last_became_peered":"2024-03-28T15:01:31.095642+0000","last_unstale":"2024-03-28T15:10:52.735802+0000","last_undegraded":"2024-03-28T15:10:52.735802+0000","last_fullsized":"2024-03-28T15:10:52.735802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'199","last_scrub_stamp":"2024-03-28T15:10:52.735720+0000","last_deep_scrub":"37'199","last_deep_scrub_stamp":"2024-03-28T15:10:52.735720+0000","last_clean_scrub_stamp":"2024-03-28T15:10:52.735720+0000","objects_scrubbed":6,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:16.590566+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":4122,"num_write":194,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21578,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"37'457","reported_seq":3298,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:53.277443+0000","last_change":"2024-03-28T15:10:53.277443+0000","last_active":"2024-03-28T15:10:53.277443+0000","last_peered":"2024-03-28T15:10:53.277443+0000","last_clean":"2024-03-28T15:10:53.277443+0000","last_became_active":"2024-03-28T15:01:32.119442+0000","last_became_peered":"2024-03-28T15:01:32.119442+0000","last_unstale":"2024-03-28T15:10:53.277443+0000","last_undegraded":"2024-03-28T15:10:53.277443+0000","last_fullsized":"2024-03-28T15:10:53.277443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'457","last_scrub_stamp":"2024-03-28T15:10:53.277320+0000","last_deep_scrub":"37'457","last_deep_scrub_stamp":"2024-03-28T15:10:53.277320+0000","last_clean_scrub_stamp":"2024-03-28T15:10:53.277320+0000","objects_scrubbed":158,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:53:57.258233+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0.001254104,"stat_sum":{"num_bytes":14752712,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":752,"num_read_kb":16498,"num_write":310,"num_write_kb":18664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"37'130","reported_seq":219,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:54.524231+0000","last_change":"2024-03-28T15:10:53.694992+0000","last_active":"2024-03-28T15:10:54.524231+0000","last_peered":"2024-03-28T15:10:54.524231+0000","last_clean":"2024-03-28T15:10:54.524231+0000","last_became_active":"2024-03-28T15:01:31.089122+0000","last_became_peered":"2024-03-28T15:01:31.089122+0000","last_unstale":"2024-03-28T15:10:54.524231+0000","last_undegraded":"2024-03-28T15:10:54.524231+0000","last_fullsized":"2024-03-28T15:10:54.524231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'129","last_scrub_stamp":"2024-03-28T15:10:53.694896+0000","last_deep_scrub":"37'129","last_deep_scrub_stamp":"2024-03-28T15:10:53.694896+0000","last_clean_scrub_stamp":"2024-03-28T15:10:53.694896+0000","objects_scrubbed":8,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:57.293264+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32804,"num_write":116,"num_write_kb":12364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14488,"num_omap_keys":27,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"37'510","reported_seq":2622,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:08.797665+0000","last_change":"2024-03-28T15:11:08.797694+0000","last_active":"2024-03-28T15:11:08.797665+0000","last_peered":"2024-03-28T15:11:08.797665+0000","last_clean":"2024-03-28T15:11:08.797665+0000","last_became_active":"2024-03-28T15:02:12.182402+0000","last_became_peered":"2024-03-28T15:02:12.182402+0000","last_unstale":"2024-03-28T15:11:08.797665+0000","last_undegraded":"2024-03-28T15:11:08.797665+0000","last_fullsized":"2024-03-28T15:11:08.797665+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'510","last_scrub_stamp":"2024-03-28T15:11:08.797594+0000","last_deep_scrub":"37'510","last_deep_scrub_stamp":"2024-03-28T15:11:08.797594+0000","last_clean_scrub_stamp":"2024-03-28T15:11:08.797594+0000","objects_scrubbed":175,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:45:54.706327+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0.0030686839999999999,"stat_sum":{"num_bytes":2787093,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":329,"num_write_kb":2901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"37'101","reported_seq":178,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:48.863213+0000","last_change":"2024-03-28T15:10:48.863213+0000","last_active":"2024-03-28T15:10:48.863213+0000","last_peered":"2024-03-28T15:10:48.863213+0000","last_clean":"2024-03-28T15:10:48.863213+0000","last_became_active":"2024-03-28T15:01:31.097866+0000","last_became_peered":"2024-03-28T15:01:31.097866+0000","last_unstale":"2024-03-28T15:10:48.863213+0000","last_undegraded":"2024-03-28T15:10:48.863213+0000","last_fullsized":"2024-03-28T15:10:48.863213+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'101","last_scrub_stamp":"2024-03-28T15:10:48.863061+0000","last_deep_scrub":"37'101","last_deep_scrub_stamp":"2024-03-28T15:10:48.863061+0000","last_clean_scrub_stamp":"2024-03-28T15:10:48.863061+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:01:03.258536+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24616,"num_write":94,"num_write_kb":8237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13508,"num_omap_keys":27,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"37'510","reported_seq":3267,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:00.892697+0000","last_change":"2024-03-28T15:11:00.892726+0000","last_active":"2024-03-28T15:11:00.892697+0000","last_peered":"2024-03-28T15:11:00.892697+0000","last_clean":"2024-03-28T15:11:00.892697+0000","last_became_active":"2024-03-28T15:02:12.180011+0000","last_became_peered":"2024-03-28T15:02:12.180011+0000","last_unstale":"2024-03-28T15:11:00.892697+0000","last_undegraded":"2024-03-28T15:11:00.892697+0000","last_fullsized":"2024-03-28T15:11:00.892697+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'510","last_scrub_stamp":"2024-03-28T15:11:00.892628+0000","last_deep_scrub":"37'510","last_deep_scrub_stamp":"2024-03-28T15:11:00.892628+0000","last_clean_scrub_stamp":"2024-03-28T15:11:00.892628+0000","objects_scrubbed":183,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:04:58.414617+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0.0018715680000000001,"stat_sum":{"num_bytes":20266861,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":674,"num_read_kb":16415,"num_write":337,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":61,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:53.244590+0000","last_change":"2024-03-28T15:10:53.244590+0000","last_active":"2024-03-28T15:10:53.244590+0000","last_peered":"2024-03-28T15:10:53.244590+0000","last_clean":"2024-03-28T15:10:53.244590+0000","last_became_active":"2024-03-28T15:01:31.090094+0000","last_became_peered":"2024-03-28T15:01:31.090094+0000","last_unstale":"2024-03-28T15:10:53.244590+0000","last_undegraded":"2024-03-28T15:10:53.244590+0000","last_fullsized":"2024-03-28T15:10:53.244590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:10:53.244502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:10:53.244502+0000","last_clean_scrub_stamp":"2024-03-28T15:10:53.244502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:12.824797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"37'527","reported_seq":2752,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:06.799338+0000","last_change":"2024-03-28T15:11:06.799396+0000","last_active":"2024-03-28T15:11:06.799338+0000","last_peered":"2024-03-28T15:11:06.799338+0000","last_clean":"2024-03-28T15:11:06.799338+0000","last_became_active":"2024-03-28T15:01:32.119527+0000","last_became_peered":"2024-03-28T15:01:32.119527+0000","last_unstale":"2024-03-28T15:11:06.799338+0000","last_undegraded":"2024-03-28T15:11:06.799338+0000","last_fullsized":"2024-03-28T15:11:06.799338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'527","last_scrub_stamp":"2024-03-28T15:11:06.799266+0000","last_deep_scrub":"37'527","last_deep_scrub_stamp":"2024-03-28T15:11:06.799266+0000","last_clean_scrub_stamp":"2024-03-28T15:11:06.799266+0000","objects_scrubbed":169,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:55.414958+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0.0022269199999999999,"stat_sum":{"num_bytes":11015620,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":136,"num_read_kb":3039,"num_write":343,"num_write_kb":10934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"37'5","reported_seq":66,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:51.621474+0000","last_change":"2024-03-28T15:10:51.621474+0000","last_active":"2024-03-28T15:10:51.621474+0000","last_peered":"2024-03-28T15:10:51.621474+0000","last_clean":"2024-03-28T15:10:51.621474+0000","last_became_active":"2024-03-28T15:01:31.088067+0000","last_became_peered":"2024-03-28T15:01:31.088067+0000","last_unstale":"2024-03-28T15:10:51.621474+0000","last_undegraded":"2024-03-28T15:10:51.621474+0000","last_fullsized":"2024-03-28T15:10:51.621474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'5","last_scrub_stamp":"2024-03-28T15:10:51.621383+0000","last_deep_scrub":"37'5","last_deep_scrub_stamp":"2024-03-28T15:10:51.621383+0000","last_clean_scrub_stamp":"2024-03-28T15:10:51.621383+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:52:44.651838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":23076,"num_omap_keys":46,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"37'493","reported_seq":3152,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:00.842046+0000","last_change":"2024-03-28T15:11:00.842085+0000","last_active":"2024-03-28T15:11:00.842046+0000","last_peered":"2024-03-28T15:11:00.842046+0000","last_clean":"2024-03-28T15:11:00.842046+0000","last_became_active":"2024-03-28T15:01:32.125910+0000","last_became_peered":"2024-03-28T15:01:32.125910+0000","last_unstale":"2024-03-28T15:11:00.842046+0000","last_undegraded":"2024-03-28T15:11:00.842046+0000","last_fullsized":"2024-03-28T15:11:00.842046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'493","last_scrub_stamp":"2024-03-28T15:11:00.841941+0000","last_deep_scrub":"37'493","last_deep_scrub_stamp":"2024-03-28T15:11:00.841941+0000","last_clean_scrub_stamp":"2024-03-28T15:11:00.841941+0000","objects_scrubbed":170,"log_size":493,"log_dups_size":0,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:01:15.237416+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0.001897143,"stat_sum":{"num_bytes":17657480,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":571,"num_read_kb":16762,"num_write":332,"num_write_kb":17420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"37'227","reported_seq":317,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:51.952875+0000","last_change":"2024-03-28T15:10:51.952875+0000","last_active":"2024-03-28T15:10:51.952875+0000","last_peered":"2024-03-28T15:10:51.952875+0000","last_clean":"2024-03-28T15:10:51.952875+0000","last_became_active":"2024-03-28T15:01:31.094234+0000","last_became_peered":"2024-03-28T15:01:31.094234+0000","last_unstale":"2024-03-28T15:10:51.952875+0000","last_undegraded":"2024-03-28T15:10:51.952875+0000","last_fullsized":"2024-03-28T15:10:51.952875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'227","last_scrub_stamp":"2024-03-28T15:10:51.952704+0000","last_deep_scrub":"37'227","last_deep_scrub_stamp":"2024-03-28T15:10:51.952704+0000","last_clean_scrub_stamp":"2024-03-28T15:10:51.952704+0000","objects_scrubbed":7,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:13:30.116660+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":83,"num_read_kb":8273,"num_write":219,"num_write_kb":8316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8570,"num_omap_keys":16,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"37'546","reported_seq":2751,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:01.867755+0000","last_change":"2024-03-28T15:11:01.867787+0000","last_active":"2024-03-28T15:11:01.867755+0000","last_peered":"2024-03-28T15:11:01.867755+0000","last_clean":"2024-03-28T15:11:01.867755+0000","last_became_active":"2024-03-28T15:01:32.113530+0000","last_became_peered":"2024-03-28T15:01:32.113530+0000","last_unstale":"2024-03-28T15:11:01.867755+0000","last_undegraded":"2024-03-28T15:11:01.867755+0000","last_fullsized":"2024-03-28T15:11:01.867755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'546","last_scrub_stamp":"2024-03-28T15:11:01.867682+0000","last_deep_scrub":"37'546","last_deep_scrub_stamp":"2024-03-28T15:11:01.867682+0000","last_clean_scrub_stamp":"2024-03-28T15:11:01.867682+0000","objects_scrubbed":186,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:31:49.897113+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0.001763492,"stat_sum":{"num_bytes":16350020,"num_objects":186,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":115,"num_read_kb":12505,"num_write":360,"num_write_kb":24351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"37'31","reported_seq":100,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:50.072753+0000","last_change":"2024-03-28T15:10:50.072804+0000","last_active":"2024-03-28T15:10:50.072753+0000","last_peered":"2024-03-28T15:10:50.072753+0000","last_clean":"2024-03-28T15:10:50.072753+0000","last_became_active":"2024-03-28T15:01:31.092135+0000","last_became_peered":"2024-03-28T15:01:31.092135+0000","last_unstale":"2024-03-28T15:10:50.072753+0000","last_undegraded":"2024-03-28T15:10:50.072753+0000","last_fullsized":"2024-03-28T15:10:50.072753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'31","last_scrub_stamp":"2024-03-28T15:10:50.072644+0000","last_deep_scrub":"37'31","last_deep_scrub_stamp":"2024-03-28T15:10:50.072644+0000","last_clean_scrub_stamp":"2024-03-28T15:10:50.072644+0000","objects_scrubbed":4,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:36:37.275129+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":29,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17912,"num_omap_keys":34,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"37'453","reported_seq":2929,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:05.824571+0000","last_change":"2024-03-28T15:11:05.824602+0000","last_active":"2024-03-28T15:11:05.824571+0000","last_peered":"2024-03-28T15:11:05.824571+0000","last_clean":"2024-03-28T15:11:05.824571+0000","last_became_active":"2024-03-28T15:01:32.116428+0000","last_became_peered":"2024-03-28T15:01:32.116428+0000","last_unstale":"2024-03-28T15:11:05.824571+0000","last_undegraded":"2024-03-28T15:11:05.824571+0000","last_fullsized":"2024-03-28T15:11:05.824571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'453","last_scrub_stamp":"2024-03-28T15:11:05.824501+0000","last_deep_scrub":"37'453","last_deep_scrub_stamp":"2024-03-28T15:11:05.824501+0000","last_clean_scrub_stamp":"2024-03-28T15:11:05.824501+0000","objects_scrubbed":163,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:03:09.847842+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0.002782082,"stat_sum":{"num_bytes":11071386,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":387,"num_read_kb":12458,"num_write":285,"num_write_kb":15052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"37'30","reported_seq":101,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:49.950785+0000","last_change":"2024-03-28T15:10:49.950813+0000","last_active":"2024-03-28T15:10:49.950785+0000","last_peered":"2024-03-28T15:10:49.950785+0000","last_clean":"2024-03-28T15:10:49.950785+0000","last_became_active":"2024-03-28T15:01:31.091918+0000","last_became_peered":"2024-03-28T15:01:31.091918+0000","last_unstale":"2024-03-28T15:10:49.950785+0000","last_undegraded":"2024-03-28T15:10:49.950785+0000","last_fullsized":"2024-03-28T15:10:49.950785+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'30","last_scrub_stamp":"2024-03-28T15:10:49.950695+0000","last_deep_scrub":"37'30","last_deep_scrub_stamp":"2024-03-28T15:10:49.950695+0000","last_clean_scrub_stamp":"2024-03-28T15:10:49.950695+0000","objects_scrubbed":2,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:00:23.610952+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":28,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"37'507","reported_seq":2885,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:03.867539+0000","last_change":"2024-03-28T15:11:03.867604+0000","last_active":"2024-03-28T15:11:03.867539+0000","last_peered":"2024-03-28T15:11:03.867539+0000","last_clean":"2024-03-28T15:11:03.867539+0000","last_became_active":"2024-03-28T15:01:32.114534+0000","last_became_peered":"2024-03-28T15:01:32.114534+0000","last_unstale":"2024-03-28T15:11:03.867539+0000","last_undegraded":"2024-03-28T15:11:03.867539+0000","last_fullsized":"2024-03-28T15:11:03.867539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'507","last_scrub_stamp":"2024-03-28T15:11:03.867463+0000","last_deep_scrub":"37'507","last_deep_scrub_stamp":"2024-03-28T15:11:03.867463+0000","last_clean_scrub_stamp":"2024-03-28T15:11:03.867463+0000","objects_scrubbed":170,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:51:59.390778+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0.0025006389999999998,"stat_sum":{"num_bytes":10266965,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":289,"num_read_kb":13000,"num_write":328,"num_write_kb":14296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"37'122","reported_seq":191,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:53.714377+0000","last_change":"2024-03-28T15:10:53.714422+0000","last_active":"2024-03-28T15:10:53.714377+0000","last_peered":"2024-03-28T15:10:53.714377+0000","last_clean":"2024-03-28T15:10:53.714377+0000","last_became_active":"2024-03-28T15:01:31.097645+0000","last_became_peered":"2024-03-28T15:01:31.097645+0000","last_unstale":"2024-03-28T15:10:53.714377+0000","last_undegraded":"2024-03-28T15:10:53.714377+0000","last_fullsized":"2024-03-28T15:10:53.714377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'122","last_scrub_stamp":"2024-03-28T15:10:53.714274+0000","last_deep_scrub":"37'122","last_deep_scrub_stamp":"2024-03-28T15:10:53.714274+0000","last_clean_scrub_stamp":"2024-03-28T15:10:53.714274+0000","objects_scrubbed":5,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:37:57.121528+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":125,"num_write_kb":4282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":40000,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"37'520","reported_seq":2839,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:01.152205+0000","last_change":"2024-03-28T15:11:01.152234+0000","last_active":"2024-03-28T15:11:01.152205+0000","last_peered":"2024-03-28T15:11:01.152205+0000","last_clean":"2024-03-28T15:11:01.152205+0000","last_became_active":"2024-03-28T15:01:32.117896+0000","last_became_peered":"2024-03-28T15:01:32.117896+0000","last_unstale":"2024-03-28T15:11:01.152205+0000","last_undegraded":"2024-03-28T15:11:01.152205+0000","last_fullsized":"2024-03-28T15:11:01.152205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'520","last_scrub_stamp":"2024-03-28T15:11:01.152136+0000","last_deep_scrub":"37'520","last_deep_scrub_stamp":"2024-03-28T15:11:01.152136+0000","last_clean_scrub_stamp":"2024-03-28T15:11:01.152136+0000","objects_scrubbed":176,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:24:54.256254+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0.003899156,"stat_sum":{"num_bytes":10797940,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":229,"num_read_kb":5058,"num_write":352,"num_write_kb":10729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"37'12","reported_seq":73,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:48.967269+0000","last_change":"2024-03-28T15:10:48.967296+0000","last_active":"2024-03-28T15:10:48.967269+0000","last_peered":"2024-03-28T15:10:48.967269+0000","last_clean":"2024-03-28T15:10:48.967269+0000","last_became_active":"2024-03-28T15:01:31.086360+0000","last_became_peered":"2024-03-28T15:01:31.086360+0000","last_unstale":"2024-03-28T15:10:48.967269+0000","last_undegraded":"2024-03-28T15:10:48.967269+0000","last_fullsized":"2024-03-28T15:10:48.967269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12","last_scrub_stamp":"2024-03-28T15:10:48.967201+0000","last_deep_scrub":"37'12","last_deep_scrub_stamp":"2024-03-28T15:10:48.967201+0000","last_clean_scrub_stamp":"2024-03-28T15:10:48.967201+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:33:34.173094+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":4136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20754,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"37'456","reported_seq":2804,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:52.854108+0000","last_change":"2024-03-28T15:10:52.854153+0000","last_active":"2024-03-28T15:10:52.854108+0000","last_peered":"2024-03-28T15:10:52.854108+0000","last_clean":"2024-03-28T15:10:52.854108+0000","last_became_active":"2024-03-28T15:01:32.112218+0000","last_became_peered":"2024-03-28T15:01:32.112218+0000","last_unstale":"2024-03-28T15:10:52.854108+0000","last_undegraded":"2024-03-28T15:10:52.854108+0000","last_fullsized":"2024-03-28T15:10:52.854108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'456","last_scrub_stamp":"2024-03-28T15:10:52.854039+0000","last_deep_scrub":"37'456","last_deep_scrub_stamp":"2024-03-28T15:10:52.854039+0000","last_clean_scrub_stamp":"2024-03-28T15:10:52.854039+0000","objects_scrubbed":146,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:59:48.658491+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0.0026010120000000002,"stat_sum":{"num_bytes":14545241,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":260,"num_read_kb":2397,"num_write":318,"num_write_kb":18466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"37'119","reported_seq":219,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:53.756498+0000","last_change":"2024-03-28T15:10:53.756525+0000","last_active":"2024-03-28T15:10:53.756498+0000","last_peered":"2024-03-28T15:10:53.756498+0000","last_clean":"2024-03-28T15:10:53.756498+0000","last_became_active":"2024-03-28T15:01:20.723010+0000","last_became_peered":"2024-03-28T15:01:20.723010+0000","last_unstale":"2024-03-28T15:10:53.756498+0000","last_undegraded":"2024-03-28T15:10:53.756498+0000","last_fullsized":"2024-03-28T15:10:53.756498+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'119","last_scrub_stamp":"2024-03-28T15:10:53.756435+0000","last_deep_scrub":"37'119","last_deep_scrub_stamp":"2024-03-28T15:10:53.756435+0000","last_clean_scrub_stamp":"2024-03-28T15:10:53.756435+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:20:19.860454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"37'87","reported_seq":149,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:50.241449+0000","last_change":"2024-03-28T15:10:50.241477+0000","last_active":"2024-03-28T15:10:50.241449+0000","last_peered":"2024-03-28T15:10:50.241449+0000","last_clean":"2024-03-28T15:10:50.241449+0000","last_became_active":"2024-03-28T15:01:31.085154+0000","last_became_peered":"2024-03-28T15:01:31.085154+0000","last_unstale":"2024-03-28T15:10:50.241449+0000","last_undegraded":"2024-03-28T15:10:50.241449+0000","last_fullsized":"2024-03-28T15:10:50.241449+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'87","last_scrub_stamp":"2024-03-28T15:10:50.241367+0000","last_deep_scrub":"37'87","last_deep_scrub_stamp":"2024-03-28T15:10:50.241367+0000","last_clean_scrub_stamp":"2024-03-28T15:10:50.241367+0000","objects_scrubbed":4,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:47.278929+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":85,"num_write_kb":4165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":28482,"num_omap_keys":54,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"37'460","reported_seq":2638,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:59.321314+0000","last_change":"2024-03-28T15:10:59.321340+0000","last_active":"2024-03-28T15:10:59.321314+0000","last_peered":"2024-03-28T15:10:59.321314+0000","last_clean":"2024-03-28T15:10:59.321314+0000","last_became_active":"2024-03-28T15:01:32.118352+0000","last_became_peered":"2024-03-28T15:01:32.118352+0000","last_unstale":"2024-03-28T15:10:59.321314+0000","last_undegraded":"2024-03-28T15:10:59.321314+0000","last_fullsized":"2024-03-28T15:10:59.321314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'460","last_scrub_stamp":"2024-03-28T15:10:59.321241+0000","last_deep_scrub":"37'460","last_deep_scrub_stamp":"2024-03-28T15:10:59.321241+0000","last_clean_scrub_stamp":"2024-03-28T15:10:59.321241+0000","objects_scrubbed":155,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:59:55.125693+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0.002201228,"stat_sum":{"num_bytes":6136325,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":89,"num_read_kb":4109,"num_write":317,"num_write_kb":10222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"37'231","reported_seq":300,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:58.813598+0000","last_change":"2024-03-28T15:10:58.813637+0000","last_active":"2024-03-28T15:10:58.813598+0000","last_peered":"2024-03-28T15:10:58.813598+0000","last_clean":"2024-03-28T15:10:58.813598+0000","last_became_active":"2024-03-28T15:01:31.095176+0000","last_became_peered":"2024-03-28T15:01:31.095176+0000","last_unstale":"2024-03-28T15:10:58.813598+0000","last_undegraded":"2024-03-28T15:10:58.813598+0000","last_fullsized":"2024-03-28T15:10:58.813598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'231","last_scrub_stamp":"2024-03-28T15:10:58.813527+0000","last_deep_scrub":"37'231","last_deep_scrub_stamp":"2024-03-28T15:10:58.813527+0000","last_clean_scrub_stamp":"2024-03-28T15:10:58.813527+0000","objects_scrubbed":8,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:14.441413+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":10,"num_read_kb":28674,"num_write":226,"num_write_kb":12578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":34058,"num_omap_keys":65,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"37'481","reported_seq":2799,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:56.914871+0000","last_change":"2024-03-28T15:10:56.914947+0000","last_active":"2024-03-28T15:10:56.914871+0000","last_peered":"2024-03-28T15:10:56.914871+0000","last_clean":"2024-03-28T15:10:56.914871+0000","last_became_active":"2024-03-28T15:01:32.122410+0000","last_became_peered":"2024-03-28T15:01:32.122410+0000","last_unstale":"2024-03-28T15:10:56.914871+0000","last_undegraded":"2024-03-28T15:10:56.914871+0000","last_fullsized":"2024-03-28T15:10:56.914871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'481","last_scrub_stamp":"2024-03-28T15:10:56.914768+0000","last_deep_scrub":"37'481","last_deep_scrub_stamp":"2024-03-28T15:10:56.914768+0000","last_clean_scrub_stamp":"2024-03-28T15:10:56.914768+0000","objects_scrubbed":173,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:12.107447+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0.0026072840000000001,"stat_sum":{"num_bytes":10172814,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":229,"num_read_kb":5026,"num_write":323,"num_write_kb":10109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'81","reported_seq":219,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:50.923665+0000","last_change":"2024-03-28T15:10:50.923690+0000","last_active":"2024-03-28T15:10:50.923665+0000","last_peered":"2024-03-28T15:10:50.923665+0000","last_clean":"2024-03-28T15:10:50.923665+0000","last_became_active":"2024-03-28T15:01:31.092980+0000","last_became_peered":"2024-03-28T15:01:31.092980+0000","last_unstale":"2024-03-28T15:10:50.923665+0000","last_undegraded":"2024-03-28T15:10:50.923665+0000","last_fullsized":"2024-03-28T15:10:50.923665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'81","last_scrub_stamp":"2024-03-28T15:10:50.923599+0000","last_deep_scrub":"37'81","last_deep_scrub_stamp":"2024-03-28T15:10:50.923599+0000","last_clean_scrub_stamp":"2024-03-28T15:10:50.923599+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:47:27.936289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534189,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":569,"num_write":78,"num_write_kb":592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15500,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"37'490","reported_seq":2597,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:54.881246+0000","last_change":"2024-03-28T15:10:54.881273+0000","last_active":"2024-03-28T15:10:54.881246+0000","last_peered":"2024-03-28T15:10:54.881246+0000","last_clean":"2024-03-28T15:10:54.881246+0000","last_became_active":"2024-03-28T15:01:32.108314+0000","last_became_peered":"2024-03-28T15:01:32.108314+0000","last_unstale":"2024-03-28T15:10:54.881246+0000","last_undegraded":"2024-03-28T15:10:54.881246+0000","last_fullsized":"2024-03-28T15:10:54.881246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'490","last_scrub_stamp":"2024-03-28T15:10:54.881165+0000","last_deep_scrub":"37'490","last_deep_scrub_stamp":"2024-03-28T15:10:54.881165+0000","last_clean_scrub_stamp":"2024-03-28T15:10:54.881165+0000","objects_scrubbed":176,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:40:42.086462+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0.0017724730000000001,"stat_sum":{"num_bytes":1382309,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":326,"num_write_kb":1524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"37'84","reported_seq":151,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:54.251559+0000","last_change":"2024-03-28T15:10:54.251602+0000","last_active":"2024-03-28T15:10:54.251559+0000","last_peered":"2024-03-28T15:10:54.251559+0000","last_clean":"2024-03-28T15:10:54.251559+0000","last_became_active":"2024-03-28T15:01:31.094472+0000","last_became_peered":"2024-03-28T15:01:31.094472+0000","last_unstale":"2024-03-28T15:10:54.251559+0000","last_undegraded":"2024-03-28T15:10:54.251559+0000","last_fullsized":"2024-03-28T15:10:54.251559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'84","last_scrub_stamp":"2024-03-28T15:10:54.251493+0000","last_deep_scrub":"37'84","last_deep_scrub_stamp":"2024-03-28T15:10:54.251493+0000","last_clean_scrub_stamp":"2024-03-28T15:10:54.251493+0000","objects_scrubbed":2,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:28:24.733010+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":80,"num_write_kb":8230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"37'501","reported_seq":2833,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:54.265544+0000","last_change":"2024-03-28T15:10:54.265594+0000","last_active":"2024-03-28T15:10:54.265544+0000","last_peered":"2024-03-28T15:10:54.265544+0000","last_clean":"2024-03-28T15:10:54.265544+0000","last_became_active":"2024-03-28T15:01:32.120335+0000","last_became_peered":"2024-03-28T15:01:32.120335+0000","last_unstale":"2024-03-28T15:10:54.265544+0000","last_undegraded":"2024-03-28T15:10:54.265544+0000","last_fullsized":"2024-03-28T15:10:54.265544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'501","last_scrub_stamp":"2024-03-28T15:10:54.265465+0000","last_deep_scrub":"37'501","last_deep_scrub_stamp":"2024-03-28T15:10:54.265465+0000","last_clean_scrub_stamp":"2024-03-28T15:10:54.265465+0000","objects_scrubbed":169,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:47:34.103156+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0.00075601100000000001,"stat_sum":{"num_bytes":5557804,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":243,"num_read_kb":4111,"num_write":349,"num_write_kb":5610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"37'125","reported_seq":188,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:49.762910+0000","last_change":"2024-03-28T15:10:49.762949+0000","last_active":"2024-03-28T15:10:49.762910+0000","last_peered":"2024-03-28T15:10:49.762910+0000","last_clean":"2024-03-28T15:10:49.762910+0000","last_became_active":"2024-03-28T15:01:31.093424+0000","last_became_peered":"2024-03-28T15:01:31.093424+0000","last_unstale":"2024-03-28T15:10:49.762910+0000","last_undegraded":"2024-03-28T15:10:49.762910+0000","last_fullsized":"2024-03-28T15:10:49.762910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'125","last_scrub_stamp":"2024-03-28T15:10:49.762835+0000","last_deep_scrub":"37'125","last_deep_scrub_stamp":"2024-03-28T15:10:49.762835+0000","last_clean_scrub_stamp":"2024-03-28T15:10:49.762835+0000","objects_scrubbed":2,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:12:22.848922+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4096,"num_write":123,"num_write_kb":4157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7532,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"37'518","reported_seq":3128,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:03.128372+0000","last_change":"2024-03-28T15:11:03.128397+0000","last_active":"2024-03-28T15:11:03.128372+0000","last_peered":"2024-03-28T15:11:03.128372+0000","last_clean":"2024-03-28T15:11:03.128372+0000","last_became_active":"2024-03-28T15:01:32.122554+0000","last_became_peered":"2024-03-28T15:01:32.122554+0000","last_unstale":"2024-03-28T15:11:03.128372+0000","last_undegraded":"2024-03-28T15:11:03.128372+0000","last_fullsized":"2024-03-28T15:11:03.128372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'518","last_scrub_stamp":"2024-03-28T15:11:03.128298+0000","last_deep_scrub":"37'518","last_deep_scrub_stamp":"2024-03-28T15:11:03.128298+0000","last_clean_scrub_stamp":"2024-03-28T15:11:03.128298+0000","objects_scrubbed":179,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:13.809814+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0.0038365610000000001,"stat_sum":{"num_bytes":18167101,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":520,"num_read_kb":17729,"num_write":341,"num_write_kb":26114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"37'1701","reported_seq":1767,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:56.820950+0000","last_change":"2024-03-28T15:10:56.820994+0000","last_active":"2024-03-28T15:10:56.820950+0000","last_peered":"2024-03-28T15:10:56.820950+0000","last_clean":"2024-03-28T15:10:56.820950+0000","last_became_active":"2024-03-28T15:01:31.094619+0000","last_became_peered":"2024-03-28T15:01:31.094619+0000","last_unstale":"2024-03-28T15:10:56.820950+0000","last_undegraded":"2024-03-28T15:10:56.820950+0000","last_fullsized":"2024-03-28T15:10:56.820950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'1701","last_scrub_stamp":"2024-03-28T15:10:56.820870+0000","last_deep_scrub":"37'1701","last_deep_scrub_stamp":"2024-03-28T15:10:56.820870+0000","last_clean_scrub_stamp":"2024-03-28T15:10:56.820870+0000","objects_scrubbed":5,"log_size":1701,"log_dups_size":0,"ondisk_log_size":1701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:31:26.134445+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":12343,"num_write":1697,"num_write_kb":9020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13956,"num_omap_keys":27,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"37'485","reported_seq":2794,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:57.934532+0000","last_change":"2024-03-28T15:10:57.934572+0000","last_active":"2024-03-28T15:10:57.934532+0000","last_peered":"2024-03-28T15:10:57.934532+0000","last_clean":"2024-03-28T15:10:57.934532+0000","last_became_active":"2024-03-28T15:01:32.111664+0000","last_became_peered":"2024-03-28T15:01:32.111664+0000","last_unstale":"2024-03-28T15:10:57.934532+0000","last_undegraded":"2024-03-28T15:10:57.934532+0000","last_fullsized":"2024-03-28T15:10:57.934532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'485","last_scrub_stamp":"2024-03-28T15:10:57.934456+0000","last_deep_scrub":"37'485","last_deep_scrub_stamp":"2024-03-28T15:10:57.934456+0000","last_clean_scrub_stamp":"2024-03-28T15:10:57.934456+0000","objects_scrubbed":166,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:53.100573+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0.0017181130000000001,"stat_sum":{"num_bytes":6397362,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":220,"num_read_kb":4361,"num_write":330,"num_write_kb":6420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'146","reported_seq":215,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:48.787578+0000","last_change":"2024-03-28T15:10:48.787629+0000","last_active":"2024-03-28T15:10:48.787578+0000","last_peered":"2024-03-28T15:10:48.787578+0000","last_clean":"2024-03-28T15:10:48.787578+0000","last_became_active":"2024-03-28T15:01:31.092912+0000","last_became_peered":"2024-03-28T15:01:31.092912+0000","last_unstale":"2024-03-28T15:10:48.787578+0000","last_undegraded":"2024-03-28T15:10:48.787578+0000","last_fullsized":"2024-03-28T15:10:48.787578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'146","last_scrub_stamp":"2024-03-28T15:10:48.787475+0000","last_deep_scrub":"37'146","last_deep_scrub_stamp":"2024-03-28T15:10:48.787475+0000","last_clean_scrub_stamp":"2024-03-28T15:10:48.787475+0000","objects_scrubbed":3,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:23:34.664993+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":239,"num_write":363,"num_write_kb":1821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":42500,"num_omap_keys":85,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"37'453","reported_seq":3313,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:56.334995+0000","last_change":"2024-03-28T15:10:56.335057+0000","last_active":"2024-03-28T15:10:56.334995+0000","last_peered":"2024-03-28T15:10:56.334995+0000","last_clean":"2024-03-28T15:10:56.334995+0000","last_became_active":"2024-03-28T15:01:32.114547+0000","last_became_peered":"2024-03-28T15:01:32.114547+0000","last_unstale":"2024-03-28T15:10:56.334995+0000","last_undegraded":"2024-03-28T15:10:56.334995+0000","last_fullsized":"2024-03-28T15:10:56.334995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'453","last_scrub_stamp":"2024-03-28T15:10:56.334917+0000","last_deep_scrub":"37'453","last_deep_scrub_stamp":"2024-03-28T15:10:56.334917+0000","last_clean_scrub_stamp":"2024-03-28T15:10:56.334917+0000","objects_scrubbed":149,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:30:50.702669+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0.002114524,"stat_sum":{"num_bytes":15625536,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":772,"num_read_kb":10204,"num_write":317,"num_write_kb":19524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"37'179","reported_seq":248,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:50.596296+0000","last_change":"2024-03-28T15:10:50.596325+0000","last_active":"2024-03-28T15:10:50.596296+0000","last_peered":"2024-03-28T15:10:50.596296+0000","last_clean":"2024-03-28T15:10:50.596296+0000","last_became_active":"2024-03-28T15:01:31.087919+0000","last_became_peered":"2024-03-28T15:01:31.087919+0000","last_unstale":"2024-03-28T15:10:50.596296+0000","last_undegraded":"2024-03-28T15:10:50.596296+0000","last_fullsized":"2024-03-28T15:10:50.596296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'179","last_scrub_stamp":"2024-03-28T15:10:50.596226+0000","last_deep_scrub":"37'179","last_deep_scrub_stamp":"2024-03-28T15:10:50.596226+0000","last_clean_scrub_stamp":"2024-03-28T15:10:50.596226+0000","objects_scrubbed":4,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:16:04.924956+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":4117,"num_write":175,"num_write_kb":4205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21520,"num_omap_keys":40,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"37'499","reported_seq":3553,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:51.021056+0000","last_change":"2024-03-28T15:10:51.021097+0000","last_active":"2024-03-28T15:10:51.021056+0000","last_peered":"2024-03-28T15:10:51.021056+0000","last_clean":"2024-03-28T15:10:51.021056+0000","last_became_active":"2024-03-28T15:01:32.114066+0000","last_became_peered":"2024-03-28T15:01:32.114066+0000","last_unstale":"2024-03-28T15:10:51.021056+0000","last_undegraded":"2024-03-28T15:10:51.021056+0000","last_fullsized":"2024-03-28T15:10:51.021056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'499","last_scrub_stamp":"2024-03-28T15:10:51.020986+0000","last_deep_scrub":"37'499","last_deep_scrub_stamp":"2024-03-28T15:10:51.020986+0000","last_clean_scrub_stamp":"2024-03-28T15:10:51.020986+0000","objects_scrubbed":170,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:38.704909+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0.0025149030000000002,"stat_sum":{"num_bytes":28522087,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":966,"num_read_kb":19238,"num_write":341,"num_write_kb":28027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'228","reported_seq":292,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:57.760346+0000","last_change":"2024-03-28T15:10:57.760415+0000","last_active":"2024-03-28T15:10:57.760346+0000","last_peered":"2024-03-28T15:10:57.760346+0000","last_clean":"2024-03-28T15:10:57.760346+0000","last_became_active":"2024-03-28T15:01:31.094732+0000","last_became_peered":"2024-03-28T15:01:31.094732+0000","last_unstale":"2024-03-28T15:10:57.760346+0000","last_undegraded":"2024-03-28T15:10:57.760346+0000","last_fullsized":"2024-03-28T15:10:57.760346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'228","last_scrub_stamp":"2024-03-28T15:10:57.760259+0000","last_deep_scrub":"37'228","last_deep_scrub_stamp":"2024-03-28T15:10:57.760259+0000","last_clean_scrub_stamp":"2024-03-28T15:10:57.760259+0000","objects_scrubbed":6,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:17:14.950543+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":230,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":47514,"num_omap_keys":92,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"37'498","reported_seq":3003,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:51.248964+0000","last_change":"2024-03-28T15:10:51.249019+0000","last_active":"2024-03-28T15:10:51.248964+0000","last_peered":"2024-03-28T15:10:51.248964+0000","last_clean":"2024-03-28T15:10:51.248964+0000","last_became_active":"2024-03-28T15:01:32.110576+0000","last_became_peered":"2024-03-28T15:01:32.110576+0000","last_unstale":"2024-03-28T15:10:51.248964+0000","last_undegraded":"2024-03-28T15:10:51.248964+0000","last_fullsized":"2024-03-28T15:10:51.248964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'498","last_scrub_stamp":"2024-03-28T15:10:51.248891+0000","last_deep_scrub":"37'498","last_deep_scrub_stamp":"2024-03-28T15:10:51.248891+0000","last_clean_scrub_stamp":"2024-03-28T15:10:51.248891+0000","objects_scrubbed":169,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:06.138497+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0.00083411100000000001,"stat_sum":{"num_bytes":27575979,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":416,"num_read_kb":10935,"num_write":349,"num_write_kb":27109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"37'21","reported_seq":83,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:51.696155+0000","last_change":"2024-03-28T15:10:51.696205+0000","last_active":"2024-03-28T15:10:51.696155+0000","last_peered":"2024-03-28T15:10:51.696155+0000","last_clean":"2024-03-28T15:10:51.696155+0000","last_became_active":"2024-03-28T15:01:31.094272+0000","last_became_peered":"2024-03-28T15:01:31.094272+0000","last_unstale":"2024-03-28T15:10:51.696155+0000","last_undegraded":"2024-03-28T15:10:51.696155+0000","last_fullsized":"2024-03-28T15:10:51.696155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'21","last_scrub_stamp":"2024-03-28T15:10:51.696056+0000","last_deep_scrub":"37'21","last_deep_scrub_stamp":"2024-03-28T15:10:51.696056+0000","last_clean_scrub_stamp":"2024-03-28T15:10:51.696056+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:46:55.294423+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":18,"num_write_kb":4204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11836,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"37'527","reported_seq":3192,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:52.309397+0000","last_change":"2024-03-28T15:10:52.309429+0000","last_active":"2024-03-28T15:10:52.309397+0000","last_peered":"2024-03-28T15:10:52.309397+0000","last_clean":"2024-03-28T15:10:52.309397+0000","last_became_active":"2024-03-28T15:01:32.111879+0000","last_became_peered":"2024-03-28T15:01:32.111879+0000","last_unstale":"2024-03-28T15:10:52.309397+0000","last_undegraded":"2024-03-28T15:10:52.309397+0000","last_fullsized":"2024-03-28T15:10:52.309397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'527","last_scrub_stamp":"2024-03-28T15:10:52.309328+0000","last_deep_scrub":"37'527","last_deep_scrub_stamp":"2024-03-28T15:10:52.309328+0000","last_clean_scrub_stamp":"2024-03-28T15:10:52.309328+0000","objects_scrubbed":183,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:48.134340+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0.0015964149999999999,"stat_sum":{"num_bytes":23553308,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":575,"num_read_kb":17168,"num_write":351,"num_write_kb":27278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"37'33","reported_seq":97,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:50.744667+0000","last_change":"2024-03-28T15:10:50.744723+0000","last_active":"2024-03-28T15:10:50.744667+0000","last_peered":"2024-03-28T15:10:50.744667+0000","last_clean":"2024-03-28T15:10:50.744667+0000","last_became_active":"2024-03-28T15:01:31.094158+0000","last_became_peered":"2024-03-28T15:01:31.094158+0000","last_unstale":"2024-03-28T15:10:50.744667+0000","last_undegraded":"2024-03-28T15:10:50.744667+0000","last_fullsized":"2024-03-28T15:10:50.744667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'33","last_scrub_stamp":"2024-03-28T15:10:50.744594+0000","last_deep_scrub":"37'33","last_deep_scrub_stamp":"2024-03-28T15:10:50.744594+0000","last_clean_scrub_stamp":"2024-03-28T15:10:50.744594+0000","objects_scrubbed":5,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:51.363441+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":31,"num_write_kb":4146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29824,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"37'474","reported_seq":3042,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:55.362985+0000","last_change":"2024-03-28T15:10:55.363016+0000","last_active":"2024-03-28T15:10:55.362985+0000","last_peered":"2024-03-28T15:10:55.362985+0000","last_clean":"2024-03-28T15:10:55.362985+0000","last_became_active":"2024-03-28T15:01:32.114923+0000","last_became_peered":"2024-03-28T15:01:32.114923+0000","last_unstale":"2024-03-28T15:10:55.362985+0000","last_undegraded":"2024-03-28T15:10:55.362985+0000","last_fullsized":"2024-03-28T15:10:55.362985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'474","last_scrub_stamp":"2024-03-28T15:10:55.362904+0000","last_deep_scrub":"37'474","last_deep_scrub_stamp":"2024-03-28T15:10:55.362904+0000","last_clean_scrub_stamp":"2024-03-28T15:10:55.362904+0000","objects_scrubbed":161,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:38.211506+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0.0022492050000000002,"stat_sum":{"num_bytes":15497791,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":479,"num_read_kb":3997,"num_write":321,"num_write_kb":15303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"37'274","reported_seq":348,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:09.391407+0000","last_change":"2024-03-28T15:10:59.768005+0000","last_active":"2024-03-28T15:11:09.391407+0000","last_peered":"2024-03-28T15:11:09.391407+0000","last_clean":"2024-03-28T15:11:09.391407+0000","last_became_active":"2024-03-28T15:01:31.096103+0000","last_became_peered":"2024-03-28T15:01:31.096103+0000","last_unstale":"2024-03-28T15:11:09.391407+0000","last_undegraded":"2024-03-28T15:11:09.391407+0000","last_fullsized":"2024-03-28T15:11:09.391407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'270","last_scrub_stamp":"2024-03-28T15:10:59.767854+0000","last_deep_scrub":"37'270","last_deep_scrub_stamp":"2024-03-28T15:10:59.767854+0000","last_clean_scrub_stamp":"2024-03-28T15:10:59.767854+0000","objects_scrubbed":7,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:45:34.976035+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10169314,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":18,"num_read_kb":28682,"num_write":265,"num_write_kb":10079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19545,"num_omap_keys":38,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"37'513","reported_seq":3014,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:57.795096+0000","last_change":"2024-03-28T15:10:57.795155+0000","last_active":"2024-03-28T15:10:57.795096+0000","last_peered":"2024-03-28T15:10:57.795096+0000","last_clean":"2024-03-28T15:10:57.795096+0000","last_became_active":"2024-03-28T15:01:32.124408+0000","last_became_peered":"2024-03-28T15:01:32.124408+0000","last_unstale":"2024-03-28T15:10:57.795096+0000","last_undegraded":"2024-03-28T15:10:57.795096+0000","last_fullsized":"2024-03-28T15:10:57.795096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'513","last_scrub_stamp":"2024-03-28T15:10:57.795003+0000","last_deep_scrub":"37'513","last_deep_scrub_stamp":"2024-03-28T15:10:57.795003+0000","last_clean_scrub_stamp":"2024-03-28T15:10:57.795003+0000","objects_scrubbed":181,"log_size":513,"log_dups_size":0,"ondisk_log_size":513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:09.377157+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0.001719216,"stat_sum":{"num_bytes":11206715,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":411,"num_read_kb":8309,"num_write":347,"num_write_kb":11133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"37'490","reported_seq":2789,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:04.137830+0000","last_change":"2024-03-28T15:11:04.137885+0000","last_active":"2024-03-28T15:11:04.137830+0000","last_peered":"2024-03-28T15:11:04.137830+0000","last_clean":"2024-03-28T15:11:04.137830+0000","last_became_active":"2024-03-28T15:01:32.123341+0000","last_became_peered":"2024-03-28T15:01:32.123341+0000","last_unstale":"2024-03-28T15:11:04.137830+0000","last_undegraded":"2024-03-28T15:11:04.137830+0000","last_fullsized":"2024-03-28T15:11:04.137830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'490","last_scrub_stamp":"2024-03-28T15:11:04.137741+0000","last_deep_scrub":"37'490","last_deep_scrub_stamp":"2024-03-28T15:11:04.137741+0000","last_clean_scrub_stamp":"2024-03-28T15:11:04.137741+0000","objects_scrubbed":163,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:19.765627+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0.0036538370000000001,"stat_sum":{"num_bytes":7862984,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":211,"num_read_kb":4119,"num_write":311,"num_write_kb":11936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"37'26","reported_seq":96,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:55.790081+0000","last_change":"2024-03-28T15:10:55.790126+0000","last_active":"2024-03-28T15:10:55.790081+0000","last_peered":"2024-03-28T15:10:55.790081+0000","last_clean":"2024-03-28T15:10:55.790081+0000","last_became_active":"2024-03-28T15:01:31.093973+0000","last_became_peered":"2024-03-28T15:01:31.093973+0000","last_unstale":"2024-03-28T15:10:55.790081+0000","last_undegraded":"2024-03-28T15:10:55.790081+0000","last_fullsized":"2024-03-28T15:10:55.790081+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'26","last_scrub_stamp":"2024-03-28T15:10:55.789986+0000","last_deep_scrub":"37'26","last_deep_scrub_stamp":"2024-03-28T15:10:55.789986+0000","last_clean_scrub_stamp":"2024-03-28T15:10:55.789986+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:24:48.894104+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":24,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"37'495","reported_seq":3846,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:56.301918+0000","last_change":"2024-03-28T15:10:56.301969+0000","last_active":"2024-03-28T15:10:56.301918+0000","last_peered":"2024-03-28T15:10:56.301918+0000","last_clean":"2024-03-28T15:10:56.301918+0000","last_became_active":"2024-03-28T15:01:32.112214+0000","last_became_peered":"2024-03-28T15:01:32.112214+0000","last_unstale":"2024-03-28T15:10:56.301918+0000","last_undegraded":"2024-03-28T15:10:56.301918+0000","last_fullsized":"2024-03-28T15:10:56.301918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'495","last_scrub_stamp":"2024-03-28T15:10:56.301816+0000","last_deep_scrub":"37'495","last_deep_scrub_stamp":"2024-03-28T15:10:56.301816+0000","last_clean_scrub_stamp":"2024-03-28T15:10:56.301816+0000","objects_scrubbed":168,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:06:32.728732+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0.0037913880000000001,"stat_sum":{"num_bytes":22932535,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":1263,"num_read_kb":23613,"num_write":335,"num_write_kb":26620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"37'175","reported_seq":251,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:54.797241+0000","last_change":"2024-03-28T15:10:54.797293+0000","last_active":"2024-03-28T15:10:54.797241+0000","last_peered":"2024-03-28T15:10:54.797241+0000","last_clean":"2024-03-28T15:10:54.797241+0000","last_became_active":"2024-03-28T15:01:31.093490+0000","last_became_peered":"2024-03-28T15:01:31.093490+0000","last_unstale":"2024-03-28T15:10:54.797241+0000","last_undegraded":"2024-03-28T15:10:54.797241+0000","last_fullsized":"2024-03-28T15:10:54.797241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'175","last_scrub_stamp":"2024-03-28T15:10:54.797133+0000","last_deep_scrub":"37'175","last_deep_scrub_stamp":"2024-03-28T15:10:54.797133+0000","last_clean_scrub_stamp":"2024-03-28T15:10:54.797133+0000","objects_scrubbed":6,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:05:45.009310+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":36885,"num_write":163,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":31608,"num_omap_keys":62,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"37'437","reported_seq":3036,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:52.916590+0000","last_change":"2024-03-28T15:10:52.916627+0000","last_active":"2024-03-28T15:10:52.916590+0000","last_peered":"2024-03-28T15:10:52.916590+0000","last_clean":"2024-03-28T15:10:52.916590+0000","last_became_active":"2024-03-28T15:01:32.117006+0000","last_became_peered":"2024-03-28T15:01:32.117006+0000","last_unstale":"2024-03-28T15:10:52.916590+0000","last_undegraded":"2024-03-28T15:10:52.916590+0000","last_fullsized":"2024-03-28T15:10:52.916590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'437","last_scrub_stamp":"2024-03-28T15:10:52.916502+0000","last_deep_scrub":"37'437","last_deep_scrub_stamp":"2024-03-28T15:10:52.916502+0000","last_clean_scrub_stamp":"2024-03-28T15:10:52.916502+0000","objects_scrubbed":149,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:59:29.346463+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0.0023175320000000002,"stat_sum":{"num_bytes":14729150,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":511,"num_read_kb":13471,"num_write":311,"num_write_kb":14544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"37'1100","reported_seq":1164,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:49.159764+0000","last_change":"2024-03-28T15:10:49.159823+0000","last_active":"2024-03-28T15:10:49.159764+0000","last_peered":"2024-03-28T15:10:49.159764+0000","last_clean":"2024-03-28T15:10:49.159764+0000","last_became_active":"2024-03-28T15:01:31.096424+0000","last_became_peered":"2024-03-28T15:01:31.096424+0000","last_unstale":"2024-03-28T15:10:49.159764+0000","last_undegraded":"2024-03-28T15:10:49.159764+0000","last_fullsized":"2024-03-28T15:10:49.159764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'1100","last_scrub_stamp":"2024-03-28T15:10:49.159683+0000","last_deep_scrub":"37'1100","last_deep_scrub_stamp":"2024-03-28T15:10:49.159683+0000","last_clean_scrub_stamp":"2024-03-28T15:10:49.159683+0000","objects_scrubbed":1,"log_size":1100,"log_dups_size":0,"ondisk_log_size":1100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:58:02.691885+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":1093,"num_write_kb":5019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"37'500","reported_seq":3089,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:11:01.373797+0000","last_change":"2024-03-28T15:11:01.373834+0000","last_active":"2024-03-28T15:11:01.373797+0000","last_peered":"2024-03-28T15:11:01.373797+0000","last_clean":"2024-03-28T15:11:01.373797+0000","last_became_active":"2024-03-28T15:01:32.119776+0000","last_became_peered":"2024-03-28T15:01:32.119776+0000","last_unstale":"2024-03-28T15:11:01.373797+0000","last_undegraded":"2024-03-28T15:11:01.373797+0000","last_fullsized":"2024-03-28T15:11:01.373797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'500","last_scrub_stamp":"2024-03-28T15:11:01.373714+0000","last_deep_scrub":"37'500","last_deep_scrub_stamp":"2024-03-28T15:11:01.373714+0000","last_clean_scrub_stamp":"2024-03-28T15:11:01.373714+0000","objects_scrubbed":167,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:17.155108+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0.0015914119999999999,"stat_sum":{"num_bytes":14776390,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":500,"num_read_kb":7280,"num_write":341,"num_write_kb":14609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"37'110","reported_seq":187,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T15:10:47.843363+0000","last_change":"2024-03-28T15:10:47.843457+0000","last_active":"2024-03-28T15:10:47.843363+0000","last_peered":"2024-03-28T15:10:47.843363+0000","last_clean":"2024-03-28T15:10:47.843363+0000","last_became_active":"2024-03-28T15:01:31.095087+0000","last_became_peered":"2024-03-28T15:01:31.095087+0000","last_unstale":"2024-03-28T15:10:47.843363+0000","last_undegraded":"2024-03-28T15:10:47.843363+0000","last_fullsized":"2024-03-28T15:10:47.843363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'110","last_scrub_stamp":"2024-03-28T15:10:47.843257+0000","last_deep_scrub":"37'110","last_deep_scrub_stamp":"2024-03-28T15:10:47.843257+0000","last_clean_scrub_stamp":"2024-03-28T15:10:47.843257+0000","objects_scrubbed":6,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:43.009374+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":20500,"num_write":104,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17374,"num_omap_keys":33,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":940897971,"num_objects":10589,"num_object_clones":0,"num_object_copies":21178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10589,"num_whiteouts":0,"num_read":30680,"num_read_kb":666406,"num_write":21173,"num_write_kb":1063512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1935458304,"data_stored":1880255846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31145,"ondisk_log_size":31145,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":182671437,"num_objects":130,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":492,"num_read_kb":424818,"num_write":6974,"num_write_kb":185001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":591173,"num_omap_keys":1136,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":365404160,"data_stored":365342874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6880,"ondisk_log_size":6880,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574970,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417524,"kb_used_data":336428,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":93954316,"statfs":{"total":96636764160,"available":96209219584,"internally_reserved":0,"allocated":344502272,"data_stored":336093657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":83032522},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574971,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302512,"kb_used_data":240808,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94069328,"statfs":{"total":96636764160,"available":96326991872,"internally_reserved":0,"allocated":246587392,"data_stored":241270326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574971,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384560,"kb_used_data":308968,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93987280,"statfs":{"total":96636764160,"available":96242974720,"internally_reserved":0,"allocated":316383232,"data_stored":308090025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574971,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414484,"kb_used_data":338892,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93957356,"statfs":{"total":96636764160,"available":96212332544,"internally_reserved":0,"allocated":347025408,"data_stored":338577860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574970,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":436444,"kb_used_data":360852,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93935396,"statfs":{"total":96636764160,"available":96189845504,"internally_reserved":0,"allocated":369512448,"data_stored":360982077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189360,"kb_used_data":149864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94182480,"statfs":{"total":96636764160,"available":96442859520,"internally_reserved":0,"allocated":153460736,"data_stored":148434971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574971,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":275412,"kb_used_data":217868,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94096428,"statfs":{"total":96636764160,"available":96354742272,"internally_reserved":0,"allocated":223096832,"data_stored":217893056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574970,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382816,"kb_used_data":297560,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93989024,"statfs":{"total":96636764160,"available":96244760576,"internally_reserved":0,"allocated":304701440,"data_stored":296790976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50331808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21508096,"data_stored":21505709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":34099200,"data_stored":34089293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":52125696,"data_stored":52112480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50332320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":73089024,"data_stored":73084456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261738496,"data_stored":254075801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172343296,"data_stored":167380119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131538944,"data_stored":126748133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334999552,"data_stored":326711655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":294531072,"data_stored":286332021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":274022400,"data_stored":265965766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":195833856,"data_stored":190756877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":270450688,"data_stored":262285474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:11:11.719 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-28T15:11:11.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.852+0000 7f0e78288640 1 Processor -- start 2024-03-28T15:11:11.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.852+0000 7f0e78288640 1 -- start start 2024-03-28T15:11:11.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e78288640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0e70154a10 0x7f0e700cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e78288640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0e700cce90 0x7f0e700cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e78288640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e7015de90 con 0x7f0e7015b2c0 2024-03-28T15:11:11.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e78288640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f0e700c5ae0 con 0x7f0e700cce90 2024-03-28T15:11:11.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e78288640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f0e7015e750 con 0x7f0e70154a10 2024-03-28T15:11:11.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e76825640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0e700cce90 0x7f0e700cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e76825640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0e700cce90 0x7f0e700cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54140/0 (socket says 172.21.15.186:54140) 2024-03-28T15:11:11.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e76825640 1 -- 172.21.15.186:0/27614610 learned_addr learned my addr 172.21.15.186:0/27614610 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:11:11.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e76024640 1 --2- 172.21.15.186:0/27614610 >> v2:172.21.15.191:3300/0 conn(0x7f0e70154a10 0x7f0e700cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e76825640 1 -- 172.21.15.186:0/27614610 >> v2:172.21.15.191:3300/0 conn(0x7f0e70154a10 msgr2=0x7f0e700cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:11:11.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e76825640 1 --2- 172.21.15.186:0/27614610 >> v2:172.21.15.191:3300/0 conn(0x7f0e70154a10 0x7f0e700cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e76825640 1 -- 172.21.15.186:0/27614610 >> v1:172.21.15.191:6789/0 conn(0x7f0e7015b2c0 legacy=0x7f0e700c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:11.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e75022640 1 -- 172.21.15.186:0/27614610 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1322880352 0 0) 0x7f0e7015de90 con 0x7f0e7015b2c0 2024-03-28T15:11:11.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e76825640 1 -- 172.21.15.186:0/27614610 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e700cb6d0 con 0x7f0e700cce90 2024-03-28T15:11:11.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e76024640 1 --2- 172.21.15.186:0/27614610 >> v2:172.21.15.191:3300/0 conn(0x7f0e70154a10 0x7f0e700cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:11:11.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.853+0000 7f0e76825640 1 --2- 172.21.15.186:0/27614610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e700cce90 0x7f0e700cd2c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0e700d1130 tx=0x7f0e6802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69512882f97148 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:11.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.855+0000 7f0e75022640 1 -- 172.21.15.186:0/27614610 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6802f850 con 0x7f0e700cce90 2024-03-28T15:11:11.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.855+0000 7f0e75022640 1 -- 172.21.15.186:0/27614610 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e6802f9f0 con 0x7f0e700cce90 2024-03-28T15:11:11.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.855+0000 7f0e75022640 1 -- 172.21.15.186:0/27614610 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6802fdc0 con 0x7f0e700cce90 2024-03-28T15:11:11.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.855+0000 7f0e78288640 1 -- 172.21.15.186:0/27614610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e700cce90 msgr2=0x7f0e700cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:11.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.855+0000 7f0e78288640 1 --2- 172.21.15.186:0/27614610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e700cce90 0x7f0e700cd2c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0e700d1130 tx=0x7f0e6802ba30 comp rx=0 tx=0).stop 2024-03-28T15:11:11.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.855+0000 7f0e78288640 1 -- 172.21.15.186:0/27614610 shutdown_connections 2024-03-28T15:11:11.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.855+0000 7f0e78288640 1 --2- 172.21.15.186:0/27614610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e700cce90 0x7f0e700cd2c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.855+0000 7f0e78288640 1 --2- 172.21.15.186:0/27614610 >> v2:172.21.15.191:3300/0 conn(0x7f0e70154a10 0x7f0e700cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.855+0000 7f0e78288640 1 -- 172.21.15.186:0/27614610 >> 172.21.15.186:0/27614610 conn(0x7f0e7006b640 msgr2=0x7f0e7006ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.855+0000 7f0e78288640 1 -- 172.21.15.186:0/27614610 shutdown_connections 2024-03-28T15:11:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.855+0000 7f0e78288640 1 -- 172.21.15.186:0/27614610 wait complete. 2024-03-28T15:11:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.856+0000 7f0e78288640 1 Processor -- start 2024-03-28T15:11:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.856+0000 7f0e78288640 1 -- start start 2024-03-28T15:11:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e78288640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e700cce90 0x7f0e7014b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e78288640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e70154a10 0x7f0e70159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e78288640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e7015b2c0 0x7f0e700cf560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e78288640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e700cb9b0 con 0x7f0e70154a10 2024-03-28T15:11:11.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e78288640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7015e3e0 con 0x7f0e700cce90 2024-03-28T15:11:11.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e78288640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f0e7015b710 con 0x7f0e7015b2c0 2024-03-28T15:11:11.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e75823640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e70154a10 0x7f0e70159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e75823640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e70154a10 0x7f0e70159d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54154/0 (socket says 172.21.15.186:54154) 2024-03-28T15:11:11.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e75823640 1 -- 172.21.15.186:0/3239078741 learned_addr learned my addr 172.21.15.186:0/3239078741 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:11:11.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e76024640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e700cce90 0x7f0e7014b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.862 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e76825640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e7015b2c0 0x7f0e700cf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.863 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e75823640 1 -- 172.21.15.186:0/3239078741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e7015b2c0 msgr2=0x7f0e700cf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:11.863 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e75823640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e7015b2c0 0x7f0e700cf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.863 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e75823640 1 -- 172.21.15.186:0/3239078741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e700cce90 msgr2=0x7f0e7014b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:11.863 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e75823640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e700cce90 0x7f0e7014b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:11.863 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.857+0000 7f0e75823640 1 -- 172.21.15.186:0/3239078741 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7014d300 con 0x7f0e70154a10 2024-03-28T15:11:11.863 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.858+0000 7f0e75823640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e70154a10 0x7f0e70159d90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000c4e0 tx=0x7f0e6000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:11.863 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.858+0000 7f0e66ffd640 1 -- 172.21.15.186:0/3239078741 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e60017070 con 0x7f0e70154a10 2024-03-28T15:11:11.863 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.858+0000 7f0e66ffd640 1 -- 172.21.15.186:0/3239078741 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e600035f0 con 0x7f0e70154a10 2024-03-28T15:11:11.863 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.858+0000 7f0e78288640 1 -- 172.21.15.186:0/3239078741 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e7014e320 con 0x7f0e70154a10 2024-03-28T15:11:11.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.858+0000 7f0e66ffd640 1 -- 172.21.15.186:0/3239078741 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e600039e0 con 0x7f0e70154a10 2024-03-28T15:11:11.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.858+0000 7f0e78288640 1 -- 172.21.15.186:0/3239078741 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e7014d570 con 0x7f0e70154a10 2024-03-28T15:11:11.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.859+0000 7f0e78288640 1 -- 172.21.15.186:0/3239078741 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e70069570 con 0x7f0e70154a10 2024-03-28T15:11:11.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.859+0000 7f0e66ffd640 1 -- 172.21.15.186:0/3239078741 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0e6001a070 con 0x7f0e70154a10 2024-03-28T15:11:11.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.860+0000 7f0e66ffd640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0e5406c560 0x7f0e5406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:11.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.860+0000 7f0e66ffd640 1 -- 172.21.15.186:0/3239078741 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f0e6008ac60 con 0x7f0e70154a10 2024-03-28T15:11:11.864 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.860+0000 7f0e76024640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0e5406c560 0x7f0e5406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:11.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.860+0000 7f0e76024640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0e5406c560 0x7f0e5406ea20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009bd0 tx=0x7f0e6c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:11.865 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:11.864+0000 7f0e66ffd640 1 -- 172.21.15.186:0/3239078741 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f0e60058980 con 0x7f0e70154a10 2024-03-28T15:11:12.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.042+0000 7f0e78288640 1 -- 172.21.15.186:0/3239078741 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f0e700cbc60 con 0x7f0e70154a10 2024-03-28T15:11:12.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.045+0000 7f0e66ffd640 1 -- 172.21.15.186:0/3239078741 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7f0e6005cc80 con 0x7f0e70154a10 2024-03-28T15:11:12.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.047+0000 7f0e66ffd640 1 -- 172.21.15.186:0/3239078741 <== mon.0 v2:172.21.15.186:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0e60061540 con 0x7f0e70154a10 2024-03-28T15:11:12.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 -- 172.21.15.186:0/3239078741 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0e5406c560 msgr2=0x7f0e5406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0e5406c560 0x7f0e5406ea20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009bd0 tx=0x7f0e6c00b040 comp rx=0 tx=0).stop 2024-03-28T15:11:12.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 -- 172.21.15.186:0/3239078741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e70154a10 msgr2=0x7f0e70159d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e70154a10 0x7f0e70159d90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000c4e0 tx=0x7f0e6000c9a0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 -- 172.21.15.186:0/3239078741 shutdown_connections 2024-03-28T15:11:12.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f0e5406c560 0x7f0e5406ea20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e7015b2c0 0x7f0e700cf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e70154a10 0x7f0e70159d90 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 --2- 172.21.15.186:0/3239078741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e700cce90 0x7f0e7014b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 -- 172.21.15.186:0/3239078741 >> 172.21.15.186:0/3239078741 conn(0x7f0e7006b640 msgr2=0x7f0e70152b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:12.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 -- 172.21.15.186:0/3239078741 shutdown_connections 2024-03-28T15:11:12.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.048+0000 7f0e78288640 1 -- 172.21.15.186:0/3239078741 wait complete. 2024-03-28T15:11:12.060 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T15:11:12.061 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:11:12.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.161+0000 7f4b746c3640 1 Processor -- start 2024-03-28T15:11:12.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.161+0000 7f4b746c3640 1 -- start start 2024-03-28T15:11:12.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b746c3640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6c150ed0 0x7f4b6c0c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b746c3640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4b6c1512b0 0x7f4b6c156d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b746c3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6c0c1b40 con 0x7f4b6c0733f0 2024-03-28T15:11:12.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b746c3640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f4b6c0c1e20 con 0x7f4b6c1512b0 2024-03-28T15:11:12.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b746c3640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6c0c18a0 con 0x7f4b6c150ed0 2024-03-28T15:11:12.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b72c60640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4b6c0733f0 0x7f4b6c0b1750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:36094/0 (socket says 172.21.15.186:36094) 2024-03-28T15:11:12.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b72c60640 1 -- 172.21.15.186:0/2211744120 learned_addr learned my addr 172.21.15.186:0/2211744120 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:11:12.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b7245f640 1 --2- 172.21.15.186:0/2211744120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6c150ed0 0x7f4b6c0c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b71c5e640 1 --2- 172.21.15.186:0/2211744120 >> v2:172.21.15.191:3300/0 conn(0x7f4b6c1512b0 0x7f4b6c156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b7145d640 1 -- 172.21.15.186:0/2211744120 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 132410344 0 0) 0x7f4b6c0c1b40 con 0x7f4b6c0733f0 2024-03-28T15:11:12.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b7145d640 1 -- 172.21.15.186:0/2211744120 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b58003520 con 0x7f4b6c0733f0 2024-03-28T15:11:12.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b7245f640 1 -- 172.21.15.186:0/2211744120 >> v2:172.21.15.191:3300/0 conn(0x7f4b6c1512b0 msgr2=0x7f4b6c156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:11:12.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b7245f640 1 --2- 172.21.15.186:0/2211744120 >> v2:172.21.15.191:3300/0 conn(0x7f4b6c1512b0 0x7f4b6c156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b7245f640 1 -- 172.21.15.186:0/2211744120 >> v1:172.21.15.186:6789/0 conn(0x7f4b6c0733f0 legacy=0x7f4b6c0b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b7245f640 1 -- 172.21.15.186:0/2211744120 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c0c15c0 con 0x7f4b6c150ed0 2024-03-28T15:11:12.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.162+0000 7f4b71c5e640 1 --2- 172.21.15.186:0/2211744120 >> v2:172.21.15.191:3300/0 conn(0x7f4b6c1512b0 0x7f4b6c156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:11:12.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.163+0000 7f4b7245f640 1 --2- 172.21.15.186:0/2211744120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6c150ed0 0x7f4b6c0c0f40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009d90 tx=0x7f4b6802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfcdb6c9d6039a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:12.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.163+0000 7f4b7145d640 1 -- 172.21.15.186:0/2211744120 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68039070 con 0x7f4b6c150ed0 2024-03-28T15:11:12.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.163+0000 7f4b7145d640 1 -- 172.21.15.186:0/2211744120 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4b6802cce0 con 0x7f4b6c150ed0 2024-03-28T15:11:12.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.163+0000 7f4b7145d640 1 -- 172.21.15.186:0/2211744120 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68034680 con 0x7f4b6c150ed0 2024-03-28T15:11:12.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.164+0000 7f4b746c3640 1 -- 172.21.15.186:0/2211744120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6c150ed0 msgr2=0x7f4b6c0c0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.164+0000 7f4b746c3640 1 --2- 172.21.15.186:0/2211744120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6c150ed0 0x7f4b6c0c0f40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009d90 tx=0x7f4b6802bf30 comp rx=0 tx=0).stop 2024-03-28T15:11:12.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.164+0000 7f4b746c3640 1 -- 172.21.15.186:0/2211744120 shutdown_connections 2024-03-28T15:11:12.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.164+0000 7f4b746c3640 1 --2- 172.21.15.186:0/2211744120 >> v2:172.21.15.191:3300/0 conn(0x7f4b6c1512b0 0x7f4b6c156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.164+0000 7f4b746c3640 1 --2- 172.21.15.186:0/2211744120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6c150ed0 0x7f4b6c0c0f40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.164+0000 7f4b746c3640 1 -- 172.21.15.186:0/2211744120 >> 172.21.15.186:0/2211744120 conn(0x7f4b6c0c8a20 msgr2=0x7f4b6c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:12.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.164+0000 7f4b746c3640 1 -- 172.21.15.186:0/2211744120 shutdown_connections 2024-03-28T15:11:12.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.164+0000 7f4b746c3640 1 -- 172.21.15.186:0/2211744120 wait complete. 2024-03-28T15:11:12.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.165+0000 7f4b746c3640 1 Processor -- start 2024-03-28T15:11:12.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.165+0000 7f4b746c3640 1 -- start start 2024-03-28T15:11:12.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b746c3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c0733f0 0x7f4b6c149530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b746c3640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6c150ed0 0x7f4b6c1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b746c3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b6c1512b0 0x7f4b6c144980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b746c3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c157290 con 0x7f4b6c0733f0 2024-03-28T15:11:12.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b746c3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c072f60 con 0x7f4b6c1512b0 2024-03-28T15:11:12.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b746c3640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6c06e5b0 con 0x7f4b6c150ed0 2024-03-28T15:11:12.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b7245f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c0733f0 0x7f4b6c149530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b7245f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c0733f0 0x7f4b6c149530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54170/0 (socket says 172.21.15.186:54170) 2024-03-28T15:11:12.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b7245f640 1 -- 172.21.15.186:0/1824323892 learned_addr learned my addr 172.21.15.186:0/1824323892 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:11:12.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b72c60640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b6c1512b0 0x7f4b6c144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b71c5e640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6c150ed0 0x7f4b6c1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b7245f640 1 -- 172.21.15.186:0/1824323892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6c150ed0 msgr2=0x7f4b6c1564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b7245f640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6c150ed0 0x7f4b6c1564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b7245f640 1 -- 172.21.15.186:0/1824323892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b6c1512b0 msgr2=0x7f4b6c144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b7245f640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b6c1512b0 0x7f4b6c144980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b7245f640 1 -- 172.21.15.186:0/1824323892 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c0cb9b0 con 0x7f4b6c0733f0 2024-03-28T15:11:12.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.166+0000 7f4b7245f640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c0733f0 0x7f4b6c149530 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009d90 tx=0x7f4b68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:12.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.167+0000 7f4b62ffd640 1 -- 172.21.15.186:0/1824323892 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68039050 con 0x7f4b6c0733f0 2024-03-28T15:11:12.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.167+0000 7f4b62ffd640 1 -- 172.21.15.186:0/1824323892 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4b68045070 con 0x7f4b6c0733f0 2024-03-28T15:11:12.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.167+0000 7f4b746c3640 1 -- 172.21.15.186:0/1824323892 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c0cbc50 con 0x7f4b6c0733f0 2024-03-28T15:11:12.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.167+0000 7f4b62ffd640 1 -- 172.21.15.186:0/1824323892 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6803e870 con 0x7f4b6c0733f0 2024-03-28T15:11:12.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.167+0000 7f4b746c3640 1 -- 172.21.15.186:0/1824323892 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6c1e2880 con 0x7f4b6c0733f0 2024-03-28T15:11:12.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.168+0000 7f4b746c3640 1 -- 172.21.15.186:0/1824323892 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b6c069500 con 0x7f4b6c0733f0 2024-03-28T15:11:12.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.168+0000 7f4b62ffd640 1 -- 172.21.15.186:0/1824323892 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4b6803ea10 con 0x7f4b6c0733f0 2024-03-28T15:11:12.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.168+0000 7f4b62ffd640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b5006c560 0x7f4b5006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.168+0000 7f4b62ffd640 1 -- 172.21.15.186:0/1824323892 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f4b680ae610 con 0x7f4b6c0733f0 2024-03-28T15:11:12.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.168+0000 7f4b71c5e640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b5006c560 0x7f4b5006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.169+0000 7f4b71c5e640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b5006c560 0x7f4b5006ea20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c071a80 tx=0x7f4b5c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:12.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.173+0000 7f4b62ffd640 1 -- 172.21.15.186:0/1824323892 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4b6807c570 con 0x7f4b6c0733f0 2024-03-28T15:11:12.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.385+0000 7f4b746c3640 1 -- 172.21.15.186:0/1824323892 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b6c0c81e0 con 0x7f4b6c0733f0 2024-03-28T15:11:12.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.385+0000 7f4b62ffd640 1 -- 172.21.15.186:0/1824323892 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6845 (secure 0 0 0) 0x7f4b68080870 con 0x7f4b6c0733f0 2024-03-28T15:11:12.386 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:11:12.386 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5407,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/249273897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":249273897},{"type":"v1","addr":"172.21.15.191:6837","nonce":249273897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:24.324220+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":25,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5278,"qdb_cluster":[5278]},"id":1}]} 2024-03-28T15:11:12.387 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 35 2024-03-28T15:11:12.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 -- 172.21.15.186:0/1824323892 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b5006c560 msgr2=0x7f4b5006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b5006c560 0x7f4b5006ea20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c071a80 tx=0x7f4b5c009190 comp rx=0 tx=0).stop 2024-03-28T15:11:12.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 -- 172.21.15.186:0/1824323892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c0733f0 msgr2=0x7f4b6c149530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c0733f0 0x7f4b6c149530 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009d90 tx=0x7f4b68004a40 comp rx=0 tx=0).stop 2024-03-28T15:11:12.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 -- 172.21.15.186:0/1824323892 shutdown_connections 2024-03-28T15:11:12.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b5006c560 0x7f4b5006ea20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b6c1512b0 0x7f4b6c144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6c150ed0 0x7f4b6c1564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 --2- 172.21.15.186:0/1824323892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b6c0733f0 0x7f4b6c149530 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 -- 172.21.15.186:0/1824323892 >> 172.21.15.186:0/1824323892 conn(0x7f4b6c0c8a20 msgr2=0x7f4b6c14e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:12.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 -- 172.21.15.186:0/1824323892 shutdown_connections 2024-03-28T15:11:12.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.388+0000 7f4b746c3640 1 -- 172.21.15.186:0/1824323892 wait complete. 2024-03-28T15:11:12.400 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:11:12.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.501+0000 7f7411120640 1 Processor -- start 2024-03-28T15:11:12.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.501+0000 7f7411120640 1 -- start start 2024-03-28T15:11:12.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f7411120640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f740c071b40 0x7f740c15ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f7411120640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f740c0c8350 0x7f740c15b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f7411120640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f740c15b8d0 con 0x7f740c154f00 2024-03-28T15:11:12.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f7411120640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f740c06f170 con 0x7f740c071b40 2024-03-28T15:11:12.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f7411120640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f740c06f450 con 0x7f740c0c8350 2024-03-28T15:11:12.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f740ad9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f740c154f00 0x7f740c06f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:36100/0 (socket says 172.21.15.186:36100) 2024-03-28T15:11:12.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f740ad9d640 1 -- 172.21.15.186:0/3541753730 learned_addr learned my addr 172.21.15.186:0/3541753730 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:11:12.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f740b59e640 1 --2- 172.21.15.186:0/3541753730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f740c0c8350 0x7f740c15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f7403fff640 1 --2- 172.21.15.186:0/3541753730 >> v2:172.21.15.191:3300/0 conn(0x7f740c071b40 0x7f740c15ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f740a59c640 1 -- 172.21.15.186:0/3541753730 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1032341134 0 0) 0x7f740c15b8d0 con 0x7f740c154f00 2024-03-28T15:11:12.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f740a59c640 1 -- 172.21.15.186:0/3541753730 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73f8003520 con 0x7f740c154f00 2024-03-28T15:11:12.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f740b59e640 1 -- 172.21.15.186:0/3541753730 >> v2:172.21.15.191:3300/0 conn(0x7f740c071b40 msgr2=0x7f740c15ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:11:12.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f740b59e640 1 --2- 172.21.15.186:0/3541753730 >> v2:172.21.15.191:3300/0 conn(0x7f740c071b40 0x7f740c15ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f740b59e640 1 -- 172.21.15.186:0/3541753730 >> v1:172.21.15.186:6789/0 conn(0x7f740c154f00 legacy=0x7f740c06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f740b59e640 1 -- 172.21.15.186:0/3541753730 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740c0c7b10 con 0x7f740c0c8350 2024-03-28T15:11:12.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.502+0000 7f7403fff640 1 --2- 172.21.15.186:0/3541753730 >> v2:172.21.15.191:3300/0 conn(0x7f740c071b40 0x7f740c15ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:11:12.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.503+0000 7f740b59e640 1 --2- 172.21.15.186:0/3541753730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f740c0c8350 0x7f740c15b070 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7404009d90 tx=0x7f740402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f46db8cccd60616 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:12.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.503+0000 7f740a59c640 1 -- 172.21.15.186:0/3541753730 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404039070 con 0x7f740c0c8350 2024-03-28T15:11:12.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.503+0000 7f740a59c640 1 -- 172.21.15.186:0/3541753730 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f740402cce0 con 0x7f740c0c8350 2024-03-28T15:11:12.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.503+0000 7f740a59c640 1 -- 172.21.15.186:0/3541753730 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404034640 con 0x7f740c0c8350 2024-03-28T15:11:12.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.504+0000 7f7411120640 1 -- 172.21.15.186:0/3541753730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f740c0c8350 msgr2=0x7f740c15b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.504+0000 7f7411120640 1 --2- 172.21.15.186:0/3541753730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f740c0c8350 0x7f740c15b070 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7404009d90 tx=0x7f740402bf30 comp rx=0 tx=0).stop 2024-03-28T15:11:12.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.504+0000 7f7411120640 1 -- 172.21.15.186:0/3541753730 shutdown_connections 2024-03-28T15:11:12.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.504+0000 7f7411120640 1 --2- 172.21.15.186:0/3541753730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f740c0c8350 0x7f740c15b070 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.504+0000 7f7411120640 1 --2- 172.21.15.186:0/3541753730 >> v2:172.21.15.191:3300/0 conn(0x7f740c071b40 0x7f740c15ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.504+0000 7f7411120640 1 -- 172.21.15.186:0/3541753730 >> 172.21.15.186:0/3541753730 conn(0x7f740c0cc580 msgr2=0x7f740c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:12.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.504+0000 7f7411120640 1 -- 172.21.15.186:0/3541753730 shutdown_connections 2024-03-28T15:11:12.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.504+0000 7f7411120640 1 -- 172.21.15.186:0/3541753730 wait complete. 2024-03-28T15:11:12.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.505+0000 7f7411120640 1 Processor -- start 2024-03-28T15:11:12.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.505+0000 7f7411120640 1 -- start start 2024-03-28T15:11:12.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f7411120640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f740c071b40 0x7f740c15dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f7411120640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f740c0c8350 0x7f740c15bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f7411120640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f740c154f00 0x7f740c1589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f7411120640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f740c15b5b0 con 0x7f740c154f00 2024-03-28T15:11:12.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f7411120640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f740c1552e0 con 0x7f740c0c8350 2024-03-28T15:11:12.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f7411120640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f740c06fff0 con 0x7f740c071b40 2024-03-28T15:11:12.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f740b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f740c154f00 0x7f740c1589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f740b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f740c154f00 0x7f740c1589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54178/0 (socket says 172.21.15.186:54178) 2024-03-28T15:11:12.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f740b59e640 1 -- 172.21.15.186:0/54320074 learned_addr learned my addr 172.21.15.186:0/54320074 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:11:12.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f740ad9d640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f740c071b40 0x7f740c15dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f7403fff640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f740c0c8350 0x7f740c15bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f740b59e640 1 -- 172.21.15.186:0/54320074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f740c071b40 msgr2=0x7f740c15dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f740b59e640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f740c071b40 0x7f740c15dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f740b59e640 1 -- 172.21.15.186:0/54320074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f740c0c8350 msgr2=0x7f740c15bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f740b59e640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f740c0c8350 0x7f740c15bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f740b59e640 1 -- 172.21.15.186:0/54320074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740c15e2a0 con 0x7f740c154f00 2024-03-28T15:11:12.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f740b59e640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f740c154f00 0x7f740c1589a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7404037040 tx=0x7f7404004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:12.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f74037fe640 1 -- 172.21.15.186:0/54320074 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404039050 con 0x7f740c154f00 2024-03-28T15:11:12.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f74037fe640 1 -- 172.21.15.186:0/54320074 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7404045070 con 0x7f740c154f00 2024-03-28T15:11:12.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f7411120640 1 -- 172.21.15.186:0/54320074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f740c1f1260 con 0x7f740c154f00 2024-03-28T15:11:12.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.506+0000 7f74037fe640 1 -- 172.21.15.186:0/54320074 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740403e850 con 0x7f740c154f00 2024-03-28T15:11:12.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.507+0000 7f7411120640 1 -- 172.21.15.186:0/54320074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f740c1f04a0 con 0x7f740c154f00 2024-03-28T15:11:12.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.508+0000 7f7411120640 1 -- 172.21.15.186:0/54320074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f740c069500 con 0x7f740c154f00 2024-03-28T15:11:12.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.508+0000 7f74037fe640 1 -- 172.21.15.186:0/54320074 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f740403e9f0 con 0x7f740c154f00 2024-03-28T15:11:12.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.508+0000 7f74037fe640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f73ec06c530 0x7f73ec06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.508+0000 7f74037fe640 1 -- 172.21.15.186:0/54320074 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f74040ae710 con 0x7f740c154f00 2024-03-28T15:11:12.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.508+0000 7f740ad9d640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f73ec06c530 0x7f73ec06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.509+0000 7f740ad9d640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f73ec06c530 0x7f73ec06e9f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f73fc005880 tx=0x7f73fc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:12.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.513+0000 7f74037fe640 1 -- 172.21.15.186:0/54320074 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f740407c550 con 0x7f740c154f00 2024-03-28T15:11:12.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.724+0000 7f7411120640 1 -- 172.21.15.186:0/54320074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f740c153330 con 0x7f740c154f00 2024-03-28T15:11:12.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.724+0000 7f74037fe640 1 -- 172.21.15.186:0/54320074 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6845 (secure 0 0 0) 0x7f7404080850 con 0x7f740c154f00 2024-03-28T15:11:12.725 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:11:12.725 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1278716016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6833","nonce":1278716016},{"type":"v1","addr":"172.21.15.191:6836","nonce":1278716016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2068761001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2068761001},{"type":"v1","addr":"172.21.15.186:6835","nonce":2068761001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2344737463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6837","nonce":2344737463},{"type":"v1","addr":"172.21.15.186:6839","nonce":2344737463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/1548903381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1548903381},{"type":"v1","addr":"172.21.15.186:6838","nonce":1548903381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5407,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/249273897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":249273897},{"type":"v1","addr":"172.21.15.191:6837","nonce":249273897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:01:31.425299+0000","modified":"2024-03-28T15:10:24.324220+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5278":{"gid":5278,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":25,"addr":"172.21.15.191:6834/3646031339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3646031339},{"type":"v1","addr":"172.21.15.191:6834","nonce":3646031339}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5278,"qdb_cluster":[5278]},"id":1}]} 2024-03-28T15:11:12.726 INFO:teuthology.orchestra.run.smithi186.stderr:dumped fsmap epoch 35 2024-03-28T15:11:12.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 -- 172.21.15.186:0/54320074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f73ec06c530 msgr2=0x7f73ec06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f73ec06c530 0x7f73ec06e9f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f73fc005880 tx=0x7f73fc009040 comp rx=0 tx=0).stop 2024-03-28T15:11:12.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 -- 172.21.15.186:0/54320074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f740c154f00 msgr2=0x7f740c1589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f740c154f00 0x7f740c1589a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7404037040 tx=0x7f7404004a40 comp rx=0 tx=0).stop 2024-03-28T15:11:12.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 -- 172.21.15.186:0/54320074 shutdown_connections 2024-03-28T15:11:12.727 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f73ec06c530 0x7f73ec06e9f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f740c154f00 0x7f740c1589a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f740c0c8350 0x7f740c15bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 --2- 172.21.15.186:0/54320074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f740c071b40 0x7f740c15dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 -- 172.21.15.186:0/54320074 >> 172.21.15.186:0/54320074 conn(0x7f740c0cc580 msgr2=0x7f740c14f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:12.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 -- 172.21.15.186:0/54320074 shutdown_connections 2024-03-28T15:11:12.728 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.726+0000 7f7411120640 1 -- 172.21.15.186:0/54320074 wait complete. 2024-03-28T15:11:12.738 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:11:12.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.839+0000 7f4e2682a640 1 Processor -- start 2024-03-28T15:11:12.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.839+0000 7f4e2682a640 1 -- start start 2024-03-28T15:11:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.839+0000 7f4e2682a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4e20071340 0x7f4e20158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.839+0000 7f4e2682a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4e200c8090 0x7f4e200c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.839+0000 7f4e2682a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e20072df0 con 0x7f4e20152810 2024-03-28T15:11:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.839+0000 7f4e2682a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f4e20159500 con 0x7f4e200c8090 2024-03-28T15:11:12.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.839+0000 7f4e2682a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f4e200c8660 con 0x7f4e20071340 2024-03-28T15:11:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.840+0000 7f4e24dc7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4e200c8090 0x7f4e200c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.840+0000 7f4e1ffff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4e20152810 0x7f4e2006c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:36114/0 (socket says 172.21.15.186:36114) 2024-03-28T15:11:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.840+0000 7f4e1ffff640 1 -- 172.21.15.186:0/522977508 learned_addr learned my addr 172.21.15.186:0/522977508 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:11:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.840+0000 7f4e1f7fe640 1 --2- 172.21.15.186:0/522977508 >> v2:172.21.15.191:3300/0 conn(0x7f4e20071340 0x7f4e20158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.840+0000 7f4e1f7fe640 1 -- 172.21.15.186:0/522977508 >> v2:172.21.15.186:3300/0 conn(0x7f4e200c8090 msgr2=0x7f4e200c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:11:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.840+0000 7f4e1f7fe640 1 --2- 172.21.15.186:0/522977508 >> v2:172.21.15.186:3300/0 conn(0x7f4e200c8090 0x7f4e200c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.840+0000 7f4e1f7fe640 1 -- 172.21.15.186:0/522977508 >> v1:172.21.15.186:6789/0 conn(0x7f4e20152810 legacy=0x7f4e2006c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.840+0000 7f4e1f7fe640 1 -- 172.21.15.186:0/522977508 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e20071d50 con 0x7f4e20071340 2024-03-28T15:11:12.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.840+0000 7f4e24dc7640 1 --2- 172.21.15.186:0/522977508 >> v2:172.21.15.186:3300/0 conn(0x7f4e200c8090 0x7f4e200c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:11:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e1f7fe640 1 --2- 172.21.15.186:0/522977508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e20071340 0x7f4e20158fc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4e200c9a60 tx=0x7f4e1002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46df96e8e6dc29a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e1effd640 1 -- 172.21.15.186:0/522977508 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10035020 con 0x7f4e20071340 2024-03-28T15:11:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e1effd640 1 -- 172.21.15.186:0/522977508 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4e10031830 con 0x7f4e20071340 2024-03-28T15:11:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e1effd640 1 -- 172.21.15.186:0/522977508 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10031c20 con 0x7f4e20071340 2024-03-28T15:11:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e2682a640 1 -- 172.21.15.186:0/522977508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e20071340 msgr2=0x7f4e20158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e2682a640 1 --2- 172.21.15.186:0/522977508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e20071340 0x7f4e20158fc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4e200c9a60 tx=0x7f4e1002b8d0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e2682a640 1 -- 172.21.15.186:0/522977508 shutdown_connections 2024-03-28T15:11:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e2682a640 1 --2- 172.21.15.186:0/522977508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e20071340 0x7f4e20158fc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e2682a640 1 --2- 172.21.15.186:0/522977508 >> v2:172.21.15.186:3300/0 conn(0x7f4e200c8090 0x7f4e200c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e2682a640 1 -- 172.21.15.186:0/522977508 >> 172.21.15.186:0/522977508 conn(0x7f4e200c8a20 msgr2=0x7f4e200c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:12.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e2682a640 1 -- 172.21.15.186:0/522977508 shutdown_connections 2024-03-28T15:11:12.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.841+0000 7f4e2682a640 1 -- 172.21.15.186:0/522977508 wait complete. 2024-03-28T15:11:12.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.843+0000 7f4e2682a640 1 Processor -- start 2024-03-28T15:11:12.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.843+0000 7f4e2682a640 1 -- start start 2024-03-28T15:11:12.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.843+0000 7f4e2682a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e20071340 0x7f4e200c23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.843+0000 7f4e2682a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e200c8090 0x7f4e200b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.843+0000 7f4e2682a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e20152810 0x7f4e200ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.843+0000 7f4e2682a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4e200725f0 con 0x7f4e20152810 2024-03-28T15:11:12.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.843+0000 7f4e2682a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4e20072310 con 0x7f4e20071340 2024-03-28T15:11:12.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.843+0000 7f4e2682a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4e200728d0 con 0x7f4e200c8090 2024-03-28T15:11:12.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e24dc7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e20152810 0x7f4e200ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e24dc7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e20152810 0x7f4e200ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54190/0 (socket says 172.21.15.186:54190) 2024-03-28T15:11:12.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e1ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e20071340 0x7f4e200c23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e200c8090 0x7f4e200b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e24dc7640 1 -- 172.21.15.186:0/4079067107 learned_addr learned my addr 172.21.15.186:0/4079067107 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:11:12.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e24dc7640 1 -- 172.21.15.186:0/4079067107 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e200c8090 msgr2=0x7f4e200b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e24dc7640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e200c8090 0x7f4e200b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e24dc7640 1 -- 172.21.15.186:0/4079067107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e20071340 msgr2=0x7f4e200c23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:12.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e24dc7640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e20071340 0x7f4e200c23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:12.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e24dc7640 1 -- 172.21.15.186:0/4079067107 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e200b03e0 con 0x7f4e20152810 2024-03-28T15:11:12.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e24dc7640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e20152810 0x7f4e200ae8e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800efc0 tx=0x7f4e1800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:12.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e1cff9640 1 -- 172.21.15.186:0/4079067107 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e18018030 con 0x7f4e20152810 2024-03-28T15:11:12.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e1cff9640 1 -- 172.21.15.186:0/4079067107 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4e1800cdf0 con 0x7f4e20152810 2024-03-28T15:11:12.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e2682a640 1 -- 172.21.15.186:0/4079067107 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e200c1070 con 0x7f4e20152810 2024-03-28T15:11:12.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e1cff9640 1 -- 172.21.15.186:0/4079067107 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e18003bd0 con 0x7f4e20152810 2024-03-28T15:11:12.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.844+0000 7f4e2682a640 1 -- 172.21.15.186:0/4079067107 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e200c1630 con 0x7f4e20152810 2024-03-28T15:11:12.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.845+0000 7f4e2682a640 1 -- 172.21.15.186:0/4079067107 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e20069500 con 0x7f4e20152810 2024-03-28T15:11:12.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.846+0000 7f4e1cff9640 1 -- 172.21.15.186:0/4079067107 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4e18005d10 con 0x7f4e20152810 2024-03-28T15:11:12.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.846+0000 7f4e1cff9640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c530 0x7f4e0c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:12.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.846+0000 7f4e1cff9640 1 -- 172.21.15.186:0/4079067107 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f4e1808c280 con 0x7f4e20152810 2024-03-28T15:11:12.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.846+0000 7f4e1ffff640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c530 0x7f4e0c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:12.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.847+0000 7f4e1ffff640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c530 0x7f4e0c06e9f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4e14007fd0 tx=0x7f4e14007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:12.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:12.850+0000 7f4e1cff9640 1 -- 172.21.15.186:0/4079067107 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4e18022070 con 0x7f4e20152810 2024-03-28T15:11:13.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.027+0000 7f4e2682a640 1 -- 172.21.15.186:0/4079067107 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e2006ce30 con 0x7f4e20152810 2024-03-28T15:11:13.029 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.029+0000 7f4e1cff9640 1 -- 172.21.15.186:0/4079067107 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+17572 (secure 0 0 0) 0x7f4e200b03e0 con 0x7f4e20152810 2024-03-28T15:11:13.029 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-28T15:11:13.030 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":37,"fsid":"694dd083-ff98-42dc-bf10-681382259053","created":"2024-03-28T15:01:06.438649+0000","modified":"2024-03-28T15:09:56.237674+0000","last_up_change":"2024-03-28T15:01:18.546153+0000","last_in_change":"2024-03-28T15:01:09.522002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:01:18.661274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:01:29.512445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:01:30.388340+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c924c36-8b44-4912-b678-273fc481ddc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6825","nonce":2448652000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6827","nonce":2448652000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6831","nonce":2448652000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2448652000},{"type":"v1","addr":"172.21.15.186:6829","nonce":2448652000}]},"public_addr":"172.21.15.186:6825/2448652000","cluster_addr":"172.21.15.186:6827/2448652000","heartbeat_back_addr":"172.21.15.186:6831/2448652000","heartbeat_front_addr":"172.21.15.186:6829/2448652000","state":["exists","up"]},{"osd":1,"uuid":"67303d31-dc97-42d2-98cf-a787593dadff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6801","nonce":2980378029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6803","nonce":2980378029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6807","nonce":2980378029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2980378029},{"type":"v1","addr":"172.21.15.186:6805","nonce":2980378029}]},"public_addr":"172.21.15.186:6801/2980378029","cluster_addr":"172.21.15.186:6803/2980378029","heartbeat_back_addr":"172.21.15.186:6807/2980378029","heartbeat_front_addr":"172.21.15.186:6805/2980378029","state":["exists","up"]},{"osd":2,"uuid":"d34a1208-ef68-48c2-9d0c-7bff414fd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6809","nonce":2659582963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6811","nonce":2659582963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6815","nonce":2659582963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2659582963},{"type":"v1","addr":"172.21.15.186:6813","nonce":2659582963}]},"public_addr":"172.21.15.186:6809/2659582963","cluster_addr":"172.21.15.186:6811/2659582963","heartbeat_back_addr":"172.21.15.186:6815/2659582963","heartbeat_front_addr":"172.21.15.186:6813/2659582963","state":["exists","up"]},{"osd":3,"uuid":"eed084b7-2749-47d1-acb3-aa2a0ff91646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6817","nonce":2463630609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6819","nonce":2463630609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6823","nonce":2463630609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2463630609},{"type":"v1","addr":"172.21.15.186:6821","nonce":2463630609}]},"public_addr":"172.21.15.186:6817/2463630609","cluster_addr":"172.21.15.186:6819/2463630609","heartbeat_back_addr":"172.21.15.186:6823/2463630609","heartbeat_front_addr":"172.21.15.186:6821/2463630609","state":["exists","up"]},{"osd":4,"uuid":"55807f69-d78a-4784-8ea5-433165104a0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6825","nonce":1698221714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6827","nonce":1698221714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6831","nonce":1698221714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1698221714},{"type":"v1","addr":"172.21.15.191:6829","nonce":1698221714}]},"public_addr":"172.21.15.191:6825/1698221714","cluster_addr":"172.21.15.191:6827/1698221714","heartbeat_back_addr":"172.21.15.191:6831/1698221714","heartbeat_front_addr":"172.21.15.191:6829/1698221714","state":["exists","up"]},{"osd":5,"uuid":"bf1fe600-cbd2-47a1-a47e-349a6c7751ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6809","nonce":138503776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6811","nonce":138503776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6815","nonce":138503776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":138503776},{"type":"v1","addr":"172.21.15.191:6813","nonce":138503776}]},"public_addr":"172.21.15.191:6809/138503776","cluster_addr":"172.21.15.191:6811/138503776","heartbeat_back_addr":"172.21.15.191:6815/138503776","heartbeat_front_addr":"172.21.15.191:6813/138503776","state":["exists","up"]},{"osd":6,"uuid":"42be8188-1e03-46e1-a6a5-8ac79b24afb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6801","nonce":1108197747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6803","nonce":1108197747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6807","nonce":1108197747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1108197747},{"type":"v1","addr":"172.21.15.191:6805","nonce":1108197747}]},"public_addr":"172.21.15.191:6801/1108197747","cluster_addr":"172.21.15.191:6803/1108197747","heartbeat_back_addr":"172.21.15.191:6807/1108197747","heartbeat_front_addr":"172.21.15.191:6805/1108197747","state":["exists","up"]},{"osd":7,"uuid":"bede2159-0ed9-4214-b776-f184cd0422d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6817","nonce":562993402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6819","nonce":562993402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6823","nonce":562993402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":562993402},{"type":"v1","addr":"172.21.15.191:6821","nonce":562993402}]},"public_addr":"172.21.15.191:6817/562993402","cluster_addr":"172.21.15.191:6819/562993402","heartbeat_back_addr":"172.21.15.191:6823/562993402","heartbeat_front_addr":"172.21.15.191:6821/562993402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.732738+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.596096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.697091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.644364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.716688+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.618324+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.720249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:01:14.662002+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.191:6834/1992706523":"2024-03-29T15:08:31.220829+0000","172.21.15.191:6837/3037586083":"2024-03-29T15:09:56.237602+0000","172.21.15.191:6835/3037586083":"2024-03-29T15:09:56.237602+0000","172.21.15.191:6837/3828930138":"2024-03-29T15:06:06.202259+0000","172.21.15.191:6835/3828930138":"2024-03-29T15:06:06.202259+0000","172.21.15.191:6832/1992706523":"2024-03-29T15:08:31.220829+0000","172.21.15.191:6834/185536353":"2024-03-29T15:03:46.182756+0000","172.21.15.191:6832/185536353":"2024-03-29T15:03:46.182756+0000","172.21.15.191:6837/2813406930":"2024-03-29T15:02:11.169402+0000","172.21.15.191:6835/2813406930":"2024-03-29T15:02:11.169402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:11:13.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 -- 172.21.15.186:0/4079067107 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c530 msgr2=0x7f4e0c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c530 0x7f4e0c06e9f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4e14007fd0 tx=0x7f4e14007e80 comp rx=0 tx=0).stop 2024-03-28T15:11:13.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 -- 172.21.15.186:0/4079067107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e20152810 msgr2=0x7f4e200ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e20152810 0x7f4e200ae8e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800efc0 tx=0x7f4e1800c490 comp rx=0 tx=0).stop 2024-03-28T15:11:13.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 -- 172.21.15.186:0/4079067107 shutdown_connections 2024-03-28T15:11:13.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4e0c06c530 0x7f4e0c06e9f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e20152810 0x7f4e200ae8e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e200c8090 0x7f4e200b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 --2- 172.21.15.186:0/4079067107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e20071340 0x7f4e200c23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 -- 172.21.15.186:0/4079067107 >> 172.21.15.186:0/4079067107 conn(0x7f4e200c8a20 msgr2=0x7f4e200c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:13.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 -- 172.21.15.186:0/4079067107 shutdown_connections 2024-03-28T15:11:13.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.031+0000 7f4e2682a640 1 -- 172.21.15.186:0/4079067107 wait complete. 2024-03-28T15:11:13.043 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T15:11:13.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.145+0000 7f48c1b5e640 1 Processor -- start 2024-03-28T15:11:13.146 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.145+0000 7f48c1b5e640 1 -- start start 2024-03-28T15:11:13.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48c1b5e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f48bc151870 0x7f48bc0c7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:13.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48c1b5e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc0c8110 0x7f48bc06ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:13.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48c1b5e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48bc0c8600 con 0x7f48bc06f120 2024-03-28T15:11:13.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48c1b5e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f48bc158070 con 0x7f48bc151870 2024-03-28T15:11:13.148 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48c1b5e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f48bc157740 con 0x7f48bc0c8110 2024-03-28T15:11:13.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48baffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc0c8110 0x7f48bc06ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:13.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48baffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc0c8110 0x7f48bc06ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.186:49564/0 (socket says 172.21.15.186:49564) 2024-03-28T15:11:13.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48baffd640 1 -- 172.21.15.186:0/286000486 learned_addr learned my addr 172.21.15.186:0/286000486 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:11:13.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48bb7fe640 1 --2- 172.21.15.186:0/286000486 >> v2:172.21.15.191:3300/0 conn(0x7f48bc151870 0x7f48bc0c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:13.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48bb7fe640 1 -- 172.21.15.186:0/286000486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc0c8110 msgr2=0x7f48bc06ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:11:13.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48bb7fe640 1 --2- 172.21.15.186:0/286000486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc0c8110 0x7f48bc06ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48bb7fe640 1 -- 172.21.15.186:0/286000486 >> v1:172.21.15.191:6789/0 conn(0x7f48bc06f120 legacy=0x7f48bc06db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48ba7fc640 1 -- 172.21.15.186:0/286000486 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2057510405 0 0) 0x7f48bc0c8600 con 0x7f48bc06f120 2024-03-28T15:11:13.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48bb7fe640 1 -- 172.21.15.186:0/286000486 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48bc157a20 con 0x7f48bc151870 2024-03-28T15:11:13.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.146+0000 7f48baffd640 1 --2- 172.21.15.186:0/286000486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc0c8110 0x7f48bc06ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:11:13.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.147+0000 7f48bb7fe640 1 --2- 172.21.15.186:0/286000486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48bc151870 0x7f48bc0c7390 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0097a0 tx=0x7f48ac02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f22f2f4626d846ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:13.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.147+0000 7f48ba7fc640 1 -- 172.21.15.186:0/286000486 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac034070 con 0x7f48bc151870 2024-03-28T15:11:13.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.147+0000 7f48ba7fc640 1 -- 172.21.15.186:0/286000486 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f48ac030610 con 0x7f48bc151870 2024-03-28T15:11:13.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.147+0000 7f48ba7fc640 1 -- 172.21.15.186:0/286000486 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac030a00 con 0x7f48bc151870 2024-03-28T15:11:13.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.148+0000 7f48c1b5e640 1 -- 172.21.15.186:0/286000486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48bc151870 msgr2=0x7f48bc0c7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.148+0000 7f48c1b5e640 1 --2- 172.21.15.186:0/286000486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48bc151870 0x7f48bc0c7390 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0097a0 tx=0x7f48ac02b960 comp rx=0 tx=0).stop 2024-03-28T15:11:13.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.148+0000 7f48c1b5e640 1 -- 172.21.15.186:0/286000486 shutdown_connections 2024-03-28T15:11:13.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.148+0000 7f48c1b5e640 1 --2- 172.21.15.186:0/286000486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc0c8110 0x7f48bc06ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.148+0000 7f48c1b5e640 1 --2- 172.21.15.186:0/286000486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48bc151870 0x7f48bc0c7390 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.148+0000 7f48c1b5e640 1 -- 172.21.15.186:0/286000486 >> 172.21.15.186:0/286000486 conn(0x7f48bc06c010 msgr2=0x7f48bc0cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:13.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.148+0000 7f48c1b5e640 1 -- 172.21.15.186:0/286000486 shutdown_connections 2024-03-28T15:11:13.151 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.148+0000 7f48c1b5e640 1 -- 172.21.15.186:0/286000486 wait complete. 2024-03-28T15:11:13.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.149+0000 7f48c1b5e640 1 Processor -- start 2024-03-28T15:11:13.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.149+0000 7f48c1b5e640 1 -- start start 2024-03-28T15:11:13.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48c1b5e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc06f120 0x7f48bc14d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:13.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48c1b5e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48bc0c8110 0x7f48bc1543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:13.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48c1b5e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48bc151870 0x7f48bc0cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:13.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48c1b5e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc0c78d0 con 0x7f48bc0c8110 2024-03-28T15:11:13.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48c1b5e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc151c50 con 0x7f48bc151870 2024-03-28T15:11:13.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48c1b5e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f48bc073e10 con 0x7f48bc06f120 2024-03-28T15:11:13.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48baffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48bc0c8110 0x7f48bc1543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:13.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48baffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48bc0c8110 0x7f48bc1543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54204/0 (socket says 172.21.15.186:54204) 2024-03-28T15:11:13.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48baffd640 1 -- 172.21.15.186:0/3899643196 learned_addr learned my addr 172.21.15.186:0/3899643196 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:11:13.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48bb7fe640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc06f120 0x7f48bc14d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:13.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48bbfff640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48bc151870 0x7f48bc0cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:13.155 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48baffd640 1 -- 172.21.15.186:0/3899643196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc06f120 msgr2=0x7f48bc14d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48baffd640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc06f120 0x7f48bc14d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48baffd640 1 -- 172.21.15.186:0/3899643196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48bc151870 msgr2=0x7f48bc0cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48baffd640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48bc151870 0x7f48bc0cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48baffd640 1 -- 172.21.15.186:0/3899643196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48bc1ec8c0 con 0x7f48bc0c8110 2024-03-28T15:11:13.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48baffd640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48bc0c8110 0x7f48bc1543f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f48b0010f40 tx=0x7f48b000e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:13.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48a3fff640 1 -- 172.21.15.186:0/3899643196 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b001e070 con 0x7f48bc0c8110 2024-03-28T15:11:13.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48a3fff640 1 -- 172.21.15.186:0/3899643196 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f48b0011860 con 0x7f48bc0c8110 2024-03-28T15:11:13.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48c1b5e640 1 -- 172.21.15.186:0/3899643196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48bc1ecba0 con 0x7f48bc0c8110 2024-03-28T15:11:13.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48a3fff640 1 -- 172.21.15.186:0/3899643196 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0011c70 con 0x7f48bc0c8110 2024-03-28T15:11:13.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.150+0000 7f48c1b5e640 1 -- 172.21.15.186:0/3899643196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48bc1ed440 con 0x7f48bc0c8110 2024-03-28T15:11:13.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.151+0000 7f48c1b5e640 1 -- 172.21.15.186:0/3899643196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48bc069570 con 0x7f48bc0c8110 2024-03-28T15:11:13.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.152+0000 7f48a3fff640 1 -- 172.21.15.186:0/3899643196 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f48b0021070 con 0x7f48bc0c8110 2024-03-28T15:11:13.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.152+0000 7f48a3fff640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48a806c530 0x7f48a806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:13.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.152+0000 7f48a3fff640 1 -- 172.21.15.186:0/3899643196 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6255+0+0 (secure 0 0 0) 0x7f48b008e320 con 0x7f48bc0c8110 2024-03-28T15:11:13.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.152+0000 7f48bb7fe640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48a806c530 0x7f48a806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:13.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.153+0000 7f48bb7fe640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48a806c530 0x7f48a806e9f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f48ac036000 tx=0x7f48ac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:13.158 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.157+0000 7f48a3fff640 1 -- 172.21.15.186:0/3899643196 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f48b005c160 con 0x7f48bc0c8110 2024-03-28T15:11:13.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.370+0000 7f48c1b5e640 1 -- 172.21.15.186:0/3899643196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f48bc154eb0 con 0x7f48bc0c8110 2024-03-28T15:11:13.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.372+0000 7f48a3fff640 1 -- 172.21.15.186:0/3899643196 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) ==== 161+0+0 (secure 0 0 0) 0x7f48b0060460 con 0x7f48bc0c8110 2024-03-28T15:11:13.372 INFO:teuthology.orchestra.run.smithi186.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T15:11:13.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 -- 172.21.15.186:0/3899643196 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48a806c530 msgr2=0x7f48a806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48a806c530 0x7f48a806e9f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f48ac036000 tx=0x7f48ac008030 comp rx=0 tx=0).stop 2024-03-28T15:11:13.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 -- 172.21.15.186:0/3899643196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48bc0c8110 msgr2=0x7f48bc1543f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48bc0c8110 0x7f48bc1543f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f48b0010f40 tx=0x7f48b000e730 comp rx=0 tx=0).stop 2024-03-28T15:11:13.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 -- 172.21.15.186:0/3899643196 shutdown_connections 2024-03-28T15:11:13.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f48a806c530 0x7f48a806e9f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48bc151870 0x7f48bc0cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48bc0c8110 0x7f48bc1543f0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 --2- 172.21.15.186:0/3899643196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48bc06f120 0x7f48bc14d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 -- 172.21.15.186:0/3899643196 >> 172.21.15.186:0/3899643196 conn(0x7f48bc06c010 msgr2=0x7f48bc072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:13.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 -- 172.21.15.186:0/3899643196 shutdown_connections 2024-03-28T15:11:13.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.374+0000 7f48c1b5e640 1 -- 172.21.15.186:0/3899643196 wait complete. 2024-03-28T15:11:13.386 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T15:11:13.416 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2024-03-28T15:11:13.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.493+0000 7f4baa0d4640 1 Processor -- start 2024-03-28T15:11:13.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.493+0000 7f4baa0d4640 1 -- start start 2024-03-28T15:11:13.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4baa0d4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ba415e620 0x7f4ba406b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:13.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4baa0d4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ba406f710 con 0x7f4ba4157fb0 2024-03-28T15:11:13.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4baa0d4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ba40b3560 con 0x7f4ba4070dd0 2024-03-28T15:11:13.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4baa0d4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba415ee00 con 0x7f4ba415e620 2024-03-28T15:11:13.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4ba37fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4ba4157fb0 0x7f4ba415df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:36120/0 (socket says 172.21.15.186:36120) 2024-03-28T15:11:13.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4ba37fe640 1 -- 172.21.15.186:0/2001782834 learned_addr learned my addr 172.21.15.186:0/2001782834 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:11:13.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4ba2ffd640 1 --2- 172.21.15.186:0/2001782834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ba415e620 0x7f4ba406b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:13.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4ba27fc640 1 -- 172.21.15.186:0/2001782834 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 824832179 0 0) 0x7f4ba406f710 con 0x7f4ba4157fb0 2024-03-28T15:11:13.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4ba27fc640 1 -- 172.21.15.186:0/2001782834 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b8c003520 con 0x7f4ba4157fb0 2024-03-28T15:11:13.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4ba27fc640 1 -- 172.21.15.186:0/2001782834 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2649923700 0 0) 0x7f4ba40b3560 con 0x7f4ba4070dd0 2024-03-28T15:11:13.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4ba27fc640 1 -- 172.21.15.186:0/2001782834 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ba406f710 con 0x7f4ba4070dd0 2024-03-28T15:11:13.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4ba2ffd640 1 -- 172.21.15.186:0/2001782834 >> v1:172.21.15.191:6789/0 conn(0x7f4ba4070dd0 legacy=0x7f4ba4165240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4ba2ffd640 1 -- 172.21.15.186:0/2001782834 >> v1:172.21.15.186:6789/0 conn(0x7f4ba4157fb0 legacy=0x7f4ba415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.494+0000 7f4ba2ffd640 1 -- 172.21.15.186:0/2001782834 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba415eb20 con 0x7f4ba415e620 2024-03-28T15:11:13.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.495+0000 7f4ba2ffd640 1 --2- 172.21.15.186:0/2001782834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ba415e620 0x7f4ba406b560 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4ba415ef80 tx=0x7f4b9802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5670f6d2cb2456b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:13.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.495+0000 7f4ba27fc640 1 -- 172.21.15.186:0/2001782834 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b98035020 con 0x7f4ba415e620 2024-03-28T15:11:13.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.495+0000 7f4ba27fc640 1 -- 172.21.15.186:0/2001782834 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4b98031890 con 0x7f4ba415e620 2024-03-28T15:11:13.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.495+0000 7f4ba27fc640 1 -- 172.21.15.186:0/2001782834 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b98031c80 con 0x7f4ba415e620 2024-03-28T15:11:13.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.496+0000 7f4baa0d4640 1 -- 172.21.15.186:0/2001782834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ba415e620 msgr2=0x7f4ba406b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.496+0000 7f4baa0d4640 1 --2- 172.21.15.186:0/2001782834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ba415e620 0x7f4ba406b560 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4ba415ef80 tx=0x7f4b9802b930 comp rx=0 tx=0).stop 2024-03-28T15:11:13.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.496+0000 7f4baa0d4640 1 -- 172.21.15.186:0/2001782834 shutdown_connections 2024-03-28T15:11:13.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.496+0000 7f4baa0d4640 1 --2- 172.21.15.186:0/2001782834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ba415e620 0x7f4ba406b560 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.496+0000 7f4baa0d4640 1 -- 172.21.15.186:0/2001782834 >> 172.21.15.186:0/2001782834 conn(0x7f4ba4073610 msgr2=0x7f4ba406d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:13.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.496+0000 7f4baa0d4640 1 -- 172.21.15.186:0/2001782834 shutdown_connections 2024-03-28T15:11:13.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.496+0000 7f4baa0d4640 1 -- 172.21.15.186:0/2001782834 wait complete. 2024-03-28T15:11:13.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.497+0000 7f4baa0d4640 1 Processor -- start 2024-03-28T15:11:13.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.497+0000 7f4baa0d4640 1 -- start start 2024-03-28T15:11:13.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4baa0d4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4070dd0 0x7f4ba415d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:13.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4baa0d4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ba4157fb0 0x7f4ba415deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:13.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4baa0d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba415e620 0x7f4ba41f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:13.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4baa0d4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba4070820 con 0x7f4ba415e620 2024-03-28T15:11:13.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4baa0d4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba406f9b0 con 0x7f4ba4070dd0 2024-03-28T15:11:13.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4baa0d4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba4158390 con 0x7f4ba4157fb0 2024-03-28T15:11:13.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4ba3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba415e620 0x7f4ba41f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:13.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4ba3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba415e620 0x7f4ba41f4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54210/0 (socket says 172.21.15.186:54210) 2024-03-28T15:11:13.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4ba3fff640 1 -- 172.21.15.186:0/4256829593 learned_addr learned my addr 172.21.15.186:0/4256829593 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:11:13.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4ba37fe640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4070dd0 0x7f4ba415d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:13.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4ba2ffd640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ba4157fb0 0x7f4ba415deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:13.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4ba3fff640 1 -- 172.21.15.186:0/4256829593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ba4157fb0 msgr2=0x7f4ba415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4ba3fff640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ba4157fb0 0x7f4ba415deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4ba3fff640 1 -- 172.21.15.186:0/4256829593 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4070dd0 msgr2=0x7f4ba415d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:13.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4ba3fff640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4070dd0 0x7f4ba415d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:13.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4ba3fff640 1 -- 172.21.15.186:0/4256829593 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba415bb00 con 0x7f4ba415e620 2024-03-28T15:11:13.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4ba3fff640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba415e620 0x7f4ba41f4100 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c6f0 tx=0x7f4b9c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:13.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4b8bfff640 1 -- 172.21.15.186:0/4256829593 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c003970 con 0x7f4ba415e620 2024-03-28T15:11:13.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4b8bfff640 1 -- 172.21.15.186:0/4256829593 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4b9c003b30 con 0x7f4ba415e620 2024-03-28T15:11:13.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4baa0d4640 1 -- 172.21.15.186:0/4256829593 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ba40cd4d0 con 0x7f4ba415e620 2024-03-28T15:11:13.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4b8bfff640 1 -- 172.21.15.186:0/4256829593 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c00a850 con 0x7f4ba415e620 2024-03-28T15:11:13.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.498+0000 7f4baa0d4640 1 -- 172.21.15.186:0/4256829593 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ba415bdc0 con 0x7f4ba415e620 2024-03-28T15:11:13.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.499+0000 7f4baa0d4640 1 -- 172.21.15.186:0/4256829593 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ba40681e0 con 0x7f4ba415e620 2024-03-28T15:11:13.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.500+0000 7f4b8bfff640 1 -- 172.21.15.186:0/4256829593 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4b9c00a9f0 con 0x7f4ba415e620 2024-03-28T15:11:13.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.500+0000 7f4b8bfff640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b9006c530 0x7f4b9006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:13.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.500+0000 7f4b8bfff640 1 -- 172.21.15.186:0/4256829593 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6341+0+0 (secure 0 0 0) 0x7f4b9c08b8a0 con 0x7f4ba415e620 2024-03-28T15:11:13.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.500+0000 7f4ba37fe640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b9006c530 0x7f4b9006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:13.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.501+0000 7f4ba37fe640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b9006c530 0x7f4b9006e9f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4b940011e0 tx=0x7f4b94009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:13.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.505+0000 7f4b8bfff640 1 -- 172.21.15.186:0/4256829593 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f4b9c059bb0 con 0x7f4ba415e620 2024-03-28T15:11:13.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:13.718+0000 7f4baa0d4640 1 -- 172.21.15.186:0/4256829593 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f4ba4069570 con 0x7f4ba415e620 2024-03-28T15:11:14.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.374+0000 7f4b8bfff640 1 -- 172.21.15.186:0/4256829593 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) ==== 104+0+0 (secure 0 0 0) 0x7f4b9c0594b0 con 0x7f4ba415e620 2024-03-28T15:11:14.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.376+0000 7f4baa0d4640 1 -- 172.21.15.186:0/4256829593 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b9006c530 msgr2=0x7f4b9006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:14.377 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.376+0000 7f4baa0d4640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b9006c530 0x7f4b9006e9f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4b940011e0 tx=0x7f4b94009040 comp rx=0 tx=0).stop 2024-03-28T15:11:14.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.376+0000 7f4baa0d4640 1 -- 172.21.15.186:0/4256829593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba415e620 msgr2=0x7f4ba41f4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:14.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.376+0000 7f4baa0d4640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba415e620 0x7f4ba41f4100 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c6f0 tx=0x7f4b9c00cbb0 comp rx=0 tx=0).stop 2024-03-28T15:11:14.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.377+0000 7f4baa0d4640 1 -- 172.21.15.186:0/4256829593 shutdown_connections 2024-03-28T15:11:14.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.377+0000 7f4baa0d4640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f4b9006c530 0x7f4b9006e9f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:14.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.377+0000 7f4baa0d4640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba415e620 0x7f4ba41f4100 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:14.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.377+0000 7f4baa0d4640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ba4157fb0 0x7f4ba415deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:14.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.377+0000 7f4baa0d4640 1 --2- 172.21.15.186:0/4256829593 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4070dd0 0x7f4ba415d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:14.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.377+0000 7f4baa0d4640 1 -- 172.21.15.186:0/4256829593 >> 172.21.15.186:0/4256829593 conn(0x7f4ba4073610 msgr2=0x7f4ba4154de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:14.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.377+0000 7f4baa0d4640 1 -- 172.21.15.186:0/4256829593 shutdown_connections 2024-03-28T15:11:14.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.377+0000 7f4baa0d4640 1 -- 172.21.15.186:0/4256829593 wait complete. 2024-03-28T15:11:14.391 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T15:11:14.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.501+0000 7fed94d81640 1 Processor -- start 2024-03-28T15:11:14.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.501+0000 7fed94d81640 1 -- start start 2024-03-28T15:11:14.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.501+0000 7fed94d81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed901515b0 0x7fed90157b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:14.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.501+0000 7fed94d81640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fed900b1d30 0x7fed90158080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:14.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.501+0000 7fed94d81640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed900c17e0 con 0x7fed900c12a0 2024-03-28T15:11:14.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.501+0000 7fed94d81640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fed90151990 con 0x7fed900b1d30 2024-03-28T15:11:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.501+0000 7fed94d81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fed9006d130 con 0x7fed901515b0 2024-03-28T15:11:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.502+0000 7fed8ed9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fed900b1d30 0x7fed90158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.502+0000 7fed87fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fed900c12a0 0x7fed900712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:36122/0 (socket says 172.21.15.186:36122) 2024-03-28T15:11:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.502+0000 7fed87fff640 1 -- 172.21.15.186:0/3555299663 learned_addr learned my addr 172.21.15.186:0/3555299663 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:11:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.502+0000 7fed8ed9d640 1 -- 172.21.15.186:0/3555299663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed901515b0 msgr2=0x7fed90157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:11:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.502+0000 7fed8e59c640 1 --2- 172.21.15.186:0/3555299663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed901515b0 0x7fed90157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.502+0000 7fed8ed9d640 1 --2- 172.21.15.186:0/3555299663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed901515b0 0x7fed90157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:14.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.502+0000 7fed8ed9d640 1 -- 172.21.15.186:0/3555299663 >> v1:172.21.15.186:6789/0 conn(0x7fed900c12a0 legacy=0x7fed900712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.502+0000 7fed8dd9b640 1 -- 172.21.15.186:0/3555299663 <== mon.0 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3480552674 0 0) 0x7fed900c17e0 con 0x7fed900c12a0 2024-03-28T15:11:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.502+0000 7fed8ed9d640 1 -- 172.21.15.186:0/3555299663 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed9006cb70 con 0x7fed900b1d30 2024-03-28T15:11:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed8ed9d640 1 --2- 172.21.15.186:0/3555299663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed900b1d30 0x7fed90158080 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fed88009d90 tx=0x7fed8802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321489a87c61bae0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed8dd9b640 1 -- 172.21.15.186:0/3555299663 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88039070 con 0x7fed900b1d30 2024-03-28T15:11:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed8dd9b640 1 -- 172.21.15.186:0/3555299663 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fed8802cce0 con 0x7fed900b1d30 2024-03-28T15:11:14.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed8dd9b640 1 -- 172.21.15.186:0/3555299663 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88034680 con 0x7fed900b1d30 2024-03-28T15:11:14.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed94d81640 1 -- 172.21.15.186:0/3555299663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed900b1d30 msgr2=0x7fed90158080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:14.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed94d81640 1 --2- 172.21.15.186:0/3555299663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed900b1d30 0x7fed90158080 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fed88009d90 tx=0x7fed8802bf30 comp rx=0 tx=0).stop 2024-03-28T15:11:14.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed94d81640 1 -- 172.21.15.186:0/3555299663 shutdown_connections 2024-03-28T15:11:14.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed94d81640 1 --2- 172.21.15.186:0/3555299663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed900b1d30 0x7fed90158080 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:14.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed94d81640 1 --2- 172.21.15.186:0/3555299663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed901515b0 0x7fed90157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:14.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed94d81640 1 -- 172.21.15.186:0/3555299663 >> 172.21.15.186:0/3555299663 conn(0x7fed900c8a20 msgr2=0x7fed900c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:14.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed94d81640 1 -- 172.21.15.186:0/3555299663 shutdown_connections 2024-03-28T15:11:14.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.503+0000 7fed94d81640 1 -- 172.21.15.186:0/3555299663 wait complete. 2024-03-28T15:11:14.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.505+0000 7fed94d81640 1 Processor -- start 2024-03-28T15:11:14.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.505+0000 7fed94d81640 1 -- start start 2024-03-28T15:11:14.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.505+0000 7fed94d81640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed900b1d30 0x7fed900b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:14.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.505+0000 7fed94d81640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed900c12a0 0x7fed9014fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:14.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.505+0000 7fed94d81640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed901515b0 0x7fed9014ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:14.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.505+0000 7fed94d81640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed90072d20 con 0x7fed900c12a0 2024-03-28T15:11:14.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.505+0000 7fed94d81640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fed9006c920 con 0x7fed901515b0 2024-03-28T15:11:14.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.505+0000 7fed94d81640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7fed900719c0 con 0x7fed900b1d30 2024-03-28T15:11:14.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.505+0000 7fed87fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed900c12a0 0x7fed9014fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:14.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.505+0000 7fed87fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed900c12a0 0x7fed9014fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54232/0 (socket says 172.21.15.186:54232) 2024-03-28T15:11:14.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.505+0000 7fed87fff640 1 -- 172.21.15.186:0/57827922 learned_addr learned my addr 172.21.15.186:0/57827922 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:11:14.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed8ed9d640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed901515b0 0x7fed9014ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:14.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed8e59c640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed900b1d30 0x7fed900b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:14.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed87fff640 1 -- 172.21.15.186:0/57827922 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed900b1d30 msgr2=0x7fed900b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:14.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed87fff640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed900b1d30 0x7fed900b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:14.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed87fff640 1 -- 172.21.15.186:0/57827922 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed901515b0 msgr2=0x7fed9014ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:14.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed87fff640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed901515b0 0x7fed9014ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:14.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed87fff640 1 -- 172.21.15.186:0/57827922 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed90156320 con 0x7fed900c12a0 2024-03-28T15:11:14.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed87fff640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed900c12a0 0x7fed9014fa70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fed7800c4d0 tx=0x7fed7800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:14.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed86ffd640 1 -- 172.21.15.186:0/57827922 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed78018070 con 0x7fed900c12a0 2024-03-28T15:11:14.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed86ffd640 1 -- 172.21.15.186:0/57827922 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fed78007500 con 0x7fed900c12a0 2024-03-28T15:11:14.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed94d81640 1 -- 172.21.15.186:0/57827922 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed90156660 con 0x7fed900c12a0 2024-03-28T15:11:14.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed86ffd640 1 -- 172.21.15.186:0/57827922 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed78004a40 con 0x7fed900c12a0 2024-03-28T15:11:14.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.506+0000 7fed94d81640 1 -- 172.21.15.186:0/57827922 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed9014f270 con 0x7fed900c12a0 2024-03-28T15:11:14.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.507+0000 7fed94d81640 1 -- 172.21.15.186:0/57827922 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed90069570 con 0x7fed900c12a0 2024-03-28T15:11:14.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.508+0000 7fed86ffd640 1 -- 172.21.15.186:0/57827922 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fed7801b070 con 0x7fed900c12a0 2024-03-28T15:11:14.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.508+0000 7fed86ffd640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fed7006c530 0x7fed7006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:14.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.508+0000 7fed86ffd640 1 -- 172.21.15.186:0/57827922 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6341+0+0 (secure 0 0 0) 0x7fed7808ad70 con 0x7fed900c12a0 2024-03-28T15:11:14.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.508+0000 7fed8e59c640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fed7006c530 0x7fed7006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:14.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.508+0000 7fed8e59c640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fed7006c530 0x7fed7006e9f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fed8000aac0 tx=0x7fed80008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:14.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.512+0000 7fed86ffd640 1 -- 172.21.15.186:0/57827922 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7fed90156660 con 0x7fed900c12a0 2024-03-28T15:11:14.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:14.689+0000 7fed94d81640 1 -- 172.21.15.186:0/57827922 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fed9014b630 con 0x7fed900c12a0 2024-03-28T15:11:15.481 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.480+0000 7fed86ffd640 1 -- 172.21.15.186:0/57827922 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v39) ==== 181+0+0 (secure 0 0 0) 0x7fed78058a30 con 0x7fed900c12a0 2024-03-28T15:11:15.481 INFO:teuthology.orchestra.run.smithi186.stderr:pool 'cephfs_metadata' removed 2024-03-28T15:11:15.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 -- 172.21.15.186:0/57827922 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fed7006c530 msgr2=0x7fed7006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:15.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fed7006c530 0x7fed7006e9f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fed8000aac0 tx=0x7fed80008040 comp rx=0 tx=0).stop 2024-03-28T15:11:15.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 -- 172.21.15.186:0/57827922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed900c12a0 msgr2=0x7fed9014fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:15.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed900c12a0 0x7fed9014fa70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fed7800c4d0 tx=0x7fed7800c990 comp rx=0 tx=0).stop 2024-03-28T15:11:15.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 -- 172.21.15.186:0/57827922 shutdown_connections 2024-03-28T15:11:15.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7fed7006c530 0x7fed7006e9f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:15.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed901515b0 0x7fed9014ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:15.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed900c12a0 0x7fed9014fa70 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:15.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 --2- 172.21.15.186:0/57827922 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed900b1d30 0x7fed900b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:15.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 -- 172.21.15.186:0/57827922 >> 172.21.15.186:0/57827922 conn(0x7fed900c8a20 msgr2=0x7fed90061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:15.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 -- 172.21.15.186:0/57827922 shutdown_connections 2024-03-28T15:11:15.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.484+0000 7fed94d81640 1 -- 172.21.15.186:0/57827922 wait complete. 2024-03-28T15:11:15.501 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T15:11:15.610 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.609+0000 7f044baa4640 1 Processor -- start 2024-03-28T15:11:15.610 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.610+0000 7f044baa4640 1 -- start start 2024-03-28T15:11:15.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.610+0000 7f044baa4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0444152aa0 0x7f04440c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:15.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f044baa4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f04440c8240 0x7f044415bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:15.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f044baa4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0444071470 con 0x7f044415c140 2024-03-28T15:11:15.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f044baa4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 -- 0x7f0444152e80 con 0x7f04440c8240 2024-03-28T15:11:15.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f044baa4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f0444071750 con 0x7f0444152aa0 2024-03-28T15:11:15.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f044903f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f04440c8240 0x7f044415bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:15.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f0449840640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0444152aa0 0x7f04440c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:15.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f044a041640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f044415c140 0x7f044406c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.186:55552/0 (socket says 172.21.15.186:55552) 2024-03-28T15:11:15.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f044a041640 1 -- 172.21.15.186:0/1356093913 learned_addr learned my addr 172.21.15.186:0/1356093913 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-28T15:11:15.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f0449840640 1 -- 172.21.15.186:0/1356093913 >> v2:172.21.15.191:3300/0 conn(0x7f04440c8240 msgr2=0x7f044415bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:11:15.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f0449840640 1 --2- 172.21.15.186:0/1356093913 >> v2:172.21.15.191:3300/0 conn(0x7f04440c8240 0x7f044415bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:15.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f0449840640 1 -- 172.21.15.186:0/1356093913 >> v1:172.21.15.191:6789/0 conn(0x7f044415c140 legacy=0x7f044406c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:15.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f0449840640 1 -- 172.21.15.186:0/1356093913 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0444072890 con 0x7f0444152aa0 2024-03-28T15:11:15.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.611+0000 7f044903f640 1 --2- 172.21.15.186:0/1356093913 >> v2:172.21.15.191:3300/0 conn(0x7f04440c8240 0x7f044415bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:11:15.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.612+0000 7f0449840640 1 --2- 172.21.15.186:0/1356093913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0444152aa0 0x7f04440c6cb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f04380097a0 tx=0x7f043802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa3c8f9352fdcdf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:15.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.612+0000 7f044883e640 1 -- 172.21.15.186:0/1356093913 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438034070 con 0x7f0444152aa0 2024-03-28T15:11:15.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.612+0000 7f044883e640 1 -- 172.21.15.186:0/1356093913 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0438030610 con 0x7f0444152aa0 2024-03-28T15:11:15.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.612+0000 7f044883e640 1 -- 172.21.15.186:0/1356093913 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438030a00 con 0x7f0444152aa0 2024-03-28T15:11:15.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.613+0000 7f044baa4640 1 -- 172.21.15.186:0/1356093913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0444152aa0 msgr2=0x7f04440c6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:15.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.613+0000 7f044baa4640 1 --2- 172.21.15.186:0/1356093913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0444152aa0 0x7f04440c6cb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f04380097a0 tx=0x7f043802b960 comp rx=0 tx=0).stop 2024-03-28T15:11:15.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.613+0000 7f044baa4640 1 -- 172.21.15.186:0/1356093913 shutdown_connections 2024-03-28T15:11:15.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.613+0000 7f044baa4640 1 --2- 172.21.15.186:0/1356093913 >> v2:172.21.15.191:3300/0 conn(0x7f04440c8240 0x7f044415bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:15.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.613+0000 7f044baa4640 1 --2- 172.21.15.186:0/1356093913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0444152aa0 0x7f04440c6cb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:15.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.613+0000 7f044baa4640 1 -- 172.21.15.186:0/1356093913 >> 172.21.15.186:0/1356093913 conn(0x7f04440c8a20 msgr2=0x7f04440c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:15.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.613+0000 7f044baa4640 1 -- 172.21.15.186:0/1356093913 shutdown_connections 2024-03-28T15:11:15.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.613+0000 7f044baa4640 1 -- 172.21.15.186:0/1356093913 wait complete. 2024-03-28T15:11:15.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.614+0000 7f044baa4640 1 Processor -- start 2024-03-28T15:11:15.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.614+0000 7f044baa4640 1 -- start start 2024-03-28T15:11:15.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f044baa4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04440c8240 0x7f04441544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:15.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f044baa4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0444152aa0 0x7f04441ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:15.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f044baa4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044415c140 0x7f04441f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:15.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f044baa4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f044406d0c0 con 0x7f04440c8240 2024-03-28T15:11:15.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f044baa4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f044415c8b0 con 0x7f044415c140 2024-03-28T15:11:15.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f044baa4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 -- 0x7f044406d240 con 0x7f0444152aa0 2024-03-28T15:11:15.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f0449840640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04440c8240 0x7f04441544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:15.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f0449840640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04440c8240 0x7f04441544e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54244/0 (socket says 172.21.15.186:54244) 2024-03-28T15:11:15.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f0449840640 1 -- 172.21.15.186:0/2702909806 learned_addr learned my addr 172.21.15.186:0/2702909806 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-28T15:11:15.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f044a041640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044415c140 0x7f04441f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:15.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f044903f640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0444152aa0 0x7f04441ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:15.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f0449840640 1 -- 172.21.15.186:0/2702909806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0444152aa0 msgr2=0x7f04441ef0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:15.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f0449840640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0444152aa0 0x7f04441ef0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:15.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f0449840640 1 -- 172.21.15.186:0/2702909806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044415c140 msgr2=0x7f04441f1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:15.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f0449840640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044415c140 0x7f04441f1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:15.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f0449840640 1 -- 172.21.15.186:0/2702909806 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04441ef630 con 0x7f04440c8240 2024-03-28T15:11:15.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f0449840640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04440c8240 0x7f04441544e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f04380098c0 tx=0x7f04380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:15.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f04367fc640 1 -- 172.21.15.186:0/2702909806 <== mon.0 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438048070 con 0x7f04440c8240 2024-03-28T15:11:15.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f04367fc640 1 -- 172.21.15.186:0/2702909806 <== mon.0 v2:172.21.15.186:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0438030430 con 0x7f04440c8240 2024-03-28T15:11:15.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f044baa4640 1 -- 172.21.15.186:0/2702909806 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0444153180 con 0x7f04440c8240 2024-03-28T15:11:15.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f04367fc640 1 -- 172.21.15.186:0/2702909806 <== mon.0 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438034040 con 0x7f04440c8240 2024-03-28T15:11:15.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.615+0000 7f044baa4640 1 -- 172.21.15.186:0/2702909806 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0444153720 con 0x7f04440c8240 2024-03-28T15:11:15.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.616+0000 7f044baa4640 1 -- 172.21.15.186:0/2702909806 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04440ca260 con 0x7f04440c8240 2024-03-28T15:11:15.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.617+0000 7f04367fc640 1 -- 172.21.15.186:0/2702909806 <== mon.0 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0438037070 con 0x7f04440c8240 2024-03-28T15:11:15.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.617+0000 7f04367fc640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f042806c560 0x7f042806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:11:15.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.617+0000 7f04367fc640 1 -- 172.21.15.186:0/2702909806 <== mon.0 v2:172.21.15.186:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 5903+0+0 (secure 0 0 0) 0x7f04380ad010 con 0x7f04440c8240 2024-03-28T15:11:15.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.617+0000 7f044903f640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f042806c560 0x7f042806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:11:15.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.618+0000 7f044903f640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f042806c560 0x7f042806ea20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0440009560 tx=0x7f0440009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:11:15.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.622+0000 7f04367fc640 1 -- 172.21.15.186:0/2702909806 <== mon.0 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186564 (secure 0 0 0) 0x7f04440ca260 con 0x7f04440c8240 2024-03-28T15:11:15.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:15.796+0000 7f044baa4640 1 -- 172.21.15.186:0/2702909806 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0444069570 con 0x7f04440c8240 2024-03-28T15:11:16.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.485+0000 7f04367fc640 1 -- 172.21.15.186:0/2702909806 <== mon.0 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v40) ==== 169+0+0 (secure 0 0 0) 0x7f04441ef630 con 0x7f04440c8240 2024-03-28T15:11:16.486 INFO:teuthology.orchestra.run.smithi186.stderr:pool 'cephfs_data' removed 2024-03-28T15:11:16.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 -- 172.21.15.186:0/2702909806 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f042806c560 msgr2=0x7f042806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:16.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f042806c560 0x7f042806ea20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0440009560 tx=0x7f0440009190 comp rx=0 tx=0).stop 2024-03-28T15:11:16.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 -- 172.21.15.186:0/2702909806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04440c8240 msgr2=0x7f04441544e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:11:16.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04440c8240 0x7f04441544e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f04380098c0 tx=0x7f04380046c0 comp rx=0 tx=0).stop 2024-03-28T15:11:16.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 -- 172.21.15.186:0/2702909806 shutdown_connections 2024-03-28T15:11:16.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.186:6832/2995751804,v1:172.21.15.186:6833/2995751804] conn(0x7f042806c560 0x7f042806ea20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:16.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044415c140 0x7f04441f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:16.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0444152aa0 0x7f04441ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:16.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 --2- 172.21.15.186:0/2702909806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04440c8240 0x7f04441544e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:11:16.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 -- 172.21.15.186:0/2702909806 >> 172.21.15.186:0/2702909806 conn(0x7f04440c8a20 msgr2=0x7f0444151b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:11:16.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 -- 172.21.15.186:0/2702909806 shutdown_connections 2024-03-28T15:11:16.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-28T15:11:16.489+0000 7f044baa4640 1 -- 172.21.15.186:0/2702909806 wait complete. 2024-03-28T15:11:16.509 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-28T15:11:16.510 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T15:11:16.510 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:16.522 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T15:11:16.522 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T15:11:16.522 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:16.573 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T15:11:16.574 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-03-28T15:11:16.574 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:16.622 INFO:tasks.ceph.mds.e:Stopped 2024-03-28T15:11:16.623 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T15:11:16.623 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:16.632 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T15:11:16.632 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T15:11:16.632 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:16.684 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T15:11:16.685 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-03-28T15:11:16.685 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:16.737 INFO:tasks.ceph.mds.f:Stopped 2024-03-28T15:11:16.737 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-28T15:11:16.738 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-28T15:11:16.738 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:16.772 INFO:tasks.ceph.osd.0:Stopped 2024-03-28T15:11:16.772 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-28T15:11:16.773 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:16.849 INFO:tasks.ceph.osd.1:Stopped 2024-03-28T15:11:16.849 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-28T15:11:16.849 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:16.931 INFO:tasks.ceph.osd.2:Stopped 2024-03-28T15:11:16.932 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-28T15:11:16.932 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:16.973 INFO:tasks.ceph.osd.3:Stopped 2024-03-28T15:11:16.973 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-28T15:11:16.974 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:17.012 INFO:tasks.ceph.osd.4:Stopped 2024-03-28T15:11:17.013 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-28T15:11:17.013 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:17.089 INFO:tasks.ceph.osd.5:Stopped 2024-03-28T15:11:17.089 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-28T15:11:17.089 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:17.161 INFO:tasks.ceph.osd.6:Stopped 2024-03-28T15:11:17.162 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-28T15:11:17.162 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:17.241 INFO:tasks.ceph.osd.7:Stopped 2024-03-28T15:11:17.241 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-28T15:11:17.241 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-28T15:11:17.241 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:17.267 INFO:tasks.ceph.mgr.y:Stopped 2024-03-28T15:11:17.267 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-28T15:11:17.267 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:17.306 INFO:tasks.ceph.mgr.x:Stopped 2024-03-28T15:11:17.306 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-28T15:11:17.306 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-28T15:11:17.306 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:17.323 INFO:tasks.ceph.mon.a:Stopped 2024-03-28T15:11:17.324 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-28T15:11:17.324 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:17.356 INFO:tasks.ceph.mon.b:Stopped 2024-03-28T15:11:17.357 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-28T15:11:17.357 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:11:17.408 INFO:tasks.ceph.mon.c:Stopped 2024-03-28T15:11:17.408 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-28T15:11:17.408 DEBUG:teuthology.orchestra.run.smithi186:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'Replacing daemon mds' | head -n 1 2024-03-28T15:11:17.450 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T15:11:17.450 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-28T15:11:18.996 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T15:11:18.996 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-28T15:11:19.097 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T15:11:19.098 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-28T15:11:19.153 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T15:11:19.153 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-28T15:11:19.213 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-28T15:11:19.213 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-28T15:11:20.224 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-28T15:11:20.225 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-28T15:11:20.321 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-28T15:11:20.322 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-28T15:11:20.379 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-28T15:11:20.379 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-28T15:11:20.440 INFO:tasks.ceph:Archiving mon data... 2024-03-28T15:11:20.441 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984/data/mon.a.tgz 2024-03-28T15:11:20.441 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-03-28T15:11:20.460 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.DWF365QeJS 2024-03-28T15:11:20.460 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.DWF365QeJS 2024-03-28T15:11:20.603 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.DWF365QeJS 2024-03-28T15:11:20.691 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.DWF365QeJS is 274KB 2024-03-28T15:11:20.786 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.DWF365QeJS 2024-03-28T15:11:20.802 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984/data/mon.b.tgz 2024-03-28T15:11:20.802 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-03-28T15:11:20.820 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.g875P65EX7 2024-03-28T15:11:20.821 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.g875P65EX7 2024-03-28T15:11:20.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.g875P65EX7 2024-03-28T15:11:21.036 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.g875P65EX7 is 278KB 2024-03-28T15:11:21.131 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.g875P65EX7 2024-03-28T15:11:21.146 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984/data/mon.c.tgz 2024-03-28T15:11:21.147 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-03-28T15:11:21.200 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.yIuTYLSKjJ 2024-03-28T15:11:21.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.yIuTYLSKjJ 2024-03-28T15:11:21.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.yIuTYLSKjJ 2024-03-28T15:11:21.401 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.yIuTYLSKjJ is 278KB 2024-03-28T15:11:21.500 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.yIuTYLSKjJ 2024-03-28T15:11:21.514 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-28T15:11:21.515 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-28T15:11:21.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-28T15:11:21.604 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-28T15:11:21.636 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-28T15:11:21.637 INFO:tasks.ceph:Archiving crash dumps... 2024-03-28T15:11:21.639 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984/remote/smithi186/crash 2024-03-28T15:11:21.640 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-28T15:11:21.676 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984/remote/smithi191/crash 2024-03-28T15:11:21.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-28T15:11:21.702 INFO:tasks.ceph:Shutting down logrotate 2024-03-28T15:11:21.703 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:11:21.721 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T15:11:21.795 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-28T15:11:21.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-28T15:11:21.852 INFO:tasks.ceph:Compressing logs... 2024-03-28T15:11:21.852 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T15:11:21.867 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T15:11:21.941 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-28T15:11:21.941 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37451.log 2024-03-28T15:11:21.942 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-28T15:11:21.942 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.37451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37611.log 2024-03-28T15:11:21.942 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.audit.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37451.log.gz 2024-03-28T15:11:21.942 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-28T15:11:21.942 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.37611.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-28T15:11:21.942 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mds.b.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37611.log.gz 2024-03-28T15:11:21.942 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37691.log 2024-03-28T15:11:21.942 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37651.log 2024-03-28T15:11:21.943 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.37691.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-28T15:11:21.943 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.37651.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37691.log.gz 2024-03-28T15:11:21.943 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 93.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-28T15:11:21.943 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-28T15:11:21.943 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37571.log 2024-03-28T15:11:21.943 INFO:teuthology.orchestra.run.smithi191.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37651.log.gz 2024-03-28T15:11:21.943 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37491.log 2024-03-28T15:11:21.943 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.37571.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-03-28T15:11:21.944 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37571.log.gz 2024-03-28T15:11:21.944 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.37491.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-28T15:11:21.944 INFO:teuthology.orchestra.run.smithi191.stderr: 88.5%/var/log/ceph/ceph-mds.f.log: -- replaced with /var/log/ceph/ceph-client.admin.37491.log.gz 2024-03-28T15:11:21.944 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-28T15:11:21.944 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37731.log 2024-03-28T15:11:21.944 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mgr.x.log: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-28T15:11:21.944 INFO:teuthology.orchestra.run.smithi191.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-28T15:11:21.944 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-28T15:11:21.944 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.37731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37531.log 2024-03-28T15:11:21.945 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37731.log.gz 2024-03-28T15:11:21.957 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-28T15:11:21.957 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.37531.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37531.log.gz 2024-03-28T15:11:21.957 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-28T15:11:21.958 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mds.d.log: 93.8% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-28T15:11:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55831.log 2024-03-28T15:11:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50115.log 2024-03-28T15:11:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54219.log 2024-03-28T15:11:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.50115.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55831.log.gz 2024-03-28T15:11:21.997 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53984.log 2024-03-28T15:11:21.998 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50115.log.gz 2024-03-28T15:11:21.998 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54219.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-28T15:11:21.998 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49131.log 2024-03-28T15:11:21.998 INFO:teuthology.orchestra.run.smithi186.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54219.log.gz 2024-03-28T15:11:21.998 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mon.a.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53984.log.gz 2024-03-28T15:11:21.998 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44805.log 2024-03-28T15:11:21.998 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56083.log 2024-03-28T15:11:21.998 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49131.log: /var/log/ceph/ceph-client.admin.44805.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.49131.log.gz -5 2024-03-28T15:11:21.999 INFO:teuthology.orchestra.run.smithi186.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43874.log 2024-03-28T15:11:21.999 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.56083.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44805.log.gz 2024-03-28T15:11:21.999 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44476.log 2024-03-28T15:11:21.999 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43874.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56083.log.gz 2024-03-28T15:11:21.999 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45848.log 2024-03-28T15:11:21.999 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43874.log.gz 2024-03-28T15:11:21.999 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47779.log 2024-03-28T15:11:21.999 INFO:teuthology.orchestra.run.smithi186.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44476.log.gz 2024-03-28T15:11:21.999 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46906.log 2024-03-28T15:11:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47779.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45848.log.gz 2024-03-28T15:11:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54046.log 2024-03-28T15:11:22.000 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47779.log.gz 2024-03-28T15:11:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.46906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55999.log 2024-03-28T15:11:22.000 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46906.log.gz 2024-03-28T15:11:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43576.log 2024-03-28T15:11:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55999.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54046.log.gz 2024-03-28T15:11:22.000 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47569.log 2024-03-28T15:11:22.000 INFO:teuthology.orchestra.run.smithi186.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55999.log.gz 2024-03-28T15:11:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44811.log 2024-03-28T15:11:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47569.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43576.log.gz 2024-03-28T15:11:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54345.log 2024-03-28T15:11:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44811.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47569.log.gz 2024-03-28T15:11:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54088.log 2024-03-28T15:11:22.001 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44811.log.gz 2024-03-28T15:11:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52451.log 2024-03-28T15:11:22.001 INFO:teuthology.orchestra.run.smithi186.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54345.log.gz 2024-03-28T15:11:22.001 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46738.log 2024-03-28T15:11:22.002 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52451.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54088.log.gzgzip 2024-03-28T15:11:22.002 INFO:teuthology.orchestra.run.smithi186.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44798.log 2024-03-28T15:11:22.002 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.46738.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52451.log.gz 2024-03-28T15:11:22.002 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56435.log 2024-03-28T15:11:22.002 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46738.log.gz 2024-03-28T15:11:22.002 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45638.log 2024-03-28T15:11:22.002 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.56435.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44798.log.gz 2024-03-28T15:11:22.002 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48134.log 2024-03-28T15:11:22.002 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56435.log.gz 2024-03-28T15:11:22.003 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43330.log 2024-03-28T15:11:22.003 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45638.log.gz 2024-03-28T15:11:22.003 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.48134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37206.log 2024-03-28T15:11:22.003 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52960.log 2024-03-28T15:11:22.003 INFO:teuthology.orchestra.run.smithi186.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48134.log.gz 2024-03-28T15:11:22.003 INFO:teuthology.orchestra.run.smithi186.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43330.log.gz 2024-03-28T15:11:22.003 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.37206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55369.log 2024-03-28T15:11:22.003 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54387.log 2024-03-28T15:11:22.004 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52960.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.37206.log.gz 2024-03-28T15:11:22.004 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43660.log 2024-03-28T15:11:22.004 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52960.log.gz 2024-03-28T15:11:22.004 INFO:teuthology.orchestra.run.smithi186.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55369.log.gz 2024-03-28T15:11:22.004 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53273.log 2024-03-28T15:11:22.004 INFO:teuthology.orchestra.run.smithi186.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54387.log.gz 2024-03-28T15:11:22.004 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45596.log 2024-03-28T15:11:22.004 INFO:teuthology.orchestra.run.smithi186.stderr: 87.6%/var/log/ceph/ceph-client.admin.53273.log: -- replaced with /var/log/ceph/ceph-client.admin.43660.log.gz 2024-03-28T15:11:22.004 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37561.log 2024-03-28T15:11:22.005 INFO:teuthology.orchestra.run.smithi186.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53273.log.gz 2024-03-28T15:11:22.005 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50013.log 2024-03-28T15:11:22.005 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.37561.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45596.log.gz 2024-03-28T15:11:22.005 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56041.log 2024-03-28T15:11:22.005 INFO:teuthology.orchestra.run.smithi186.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37561.log.gz 2024-03-28T15:11:22.005 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.50013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52535.log 2024-03-28T15:11:22.005 INFO:teuthology.orchestra.run.smithi186.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50013.log.gz 2024-03-28T15:11:22.005 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.56041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55579.log 2024-03-28T15:11:22.005 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52535.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56041.log.gz 2024-03-28T15:11:22.006 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49236.log 2024-03-28T15:11:22.006 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55579.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52535.log.gz 2024-03-28T15:11:22.006 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55285.log 2024-03-28T15:11:22.006 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55579.log.gz 2024-03-28T15:11:22.006 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43746.log 2024-03-28T15:11:22.006 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55285.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.49236.log.gz 2024-03-28T15:11:22.006 INFO:teuthology.orchestra.run.smithi186.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52918.log 2024-03-28T15:11:22.006 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43746.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55285.log.gz 2024-03-28T15:11:22.007 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56351.log 2024-03-28T15:11:22.007 INFO:teuthology.orchestra.run.smithi186.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43746.log.gz 2024-03-28T15:11:22.007 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52155.log 2024-03-28T15:11:22.007 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.56351.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52918.log.gz 2024-03-28T15:11:22.007 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44812.log 2024-03-28T15:11:22.007 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56351.log.gz 2024-03-28T15:11:22.007 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45806.log 2024-03-28T15:11:22.007 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52155.log.gz 2024-03-28T15:11:22.007 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51842.log 2024-03-28T15:11:22.008 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45806.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44812.log.gz 2024-03-28T15:11:22.008 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46076.log 2024-03-28T15:11:22.008 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.51842.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45806.log.gz 2024-03-28T15:11:22.008 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49194.log 2024-03-28T15:11:22.008 INFO:teuthology.orchestra.run.smithi186.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51842.log.gz 2024-03-28T15:11:22.008 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.46076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44472.log 2024-03-28T15:11:22.008 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49194.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46076.log.gz 2024-03-28T15:11:22.008 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53002.log 2024-03-28T15:11:22.008 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44472.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49194.log.gz 2024-03-28T15:11:22.009 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43916.log 2024-03-28T15:11:22.009 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44472.log.gz 2024-03-28T15:11:22.009 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43534.log 2024-03-28T15:11:22.009 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43916.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53002.log.gz 2024-03-28T15:11:22.009 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55453.log 2024-03-28T15:11:22.009 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43534.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43916.log.gz 2024-03-28T15:11:22.009 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52366.log 2024-03-28T15:11:22.009 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43534.log.gz 2024-03-28T15:11:22.010 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56519.log 2024-03-28T15:11:22.010 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52366.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55453.log.gz 2024-03-28T15:11:22.010 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46780.log 2024-03-28T15:11:22.010 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7%/var/log/ceph/ceph-client.admin.56519.log: -- replaced with /var/log/ceph/ceph-client.admin.52366.log.gz 2024-03-28T15:11:22.010 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44042.log 2024-03-28T15:11:22.010 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56519.log.gz 2024-03-28T15:11:22.010 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.46780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49929.log 2024-03-28T15:11:22.010 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44042.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46780.log.gz 2024-03-28T15:11:22.010 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47074.log 2024-03-28T15:11:22.011 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49929.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44042.log.gz 2024-03-28T15:11:22.011 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45680.log 2024-03-28T15:11:22.011 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49929.log.gz 2024-03-28T15:11:22.011 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55017.log 2024-03-28T15:11:22.011 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45680.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47074.log.gz 2024-03-28T15:11:22.011 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44210.log 2024-03-28T15:11:22.011 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45680.log.gz 2024-03-28T15:11:22.011 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55747.log 2024-03-28T15:11:22.012 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55017.log.gz 2024-03-28T15:11:22.012 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46822.log 2024-03-28T15:11:22.012 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55747.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44210.log.gz 2024-03-28T15:11:22.012 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44468.log 2024-03-28T15:11:22.012 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.46822.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55747.log.gz 2024-03-28T15:11:22.012 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51969.log 2024-03-28T15:11:22.012 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44468.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46822.log.gz 2024-03-28T15:11:22.012 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47242.log 2024-03-28T15:11:22.012 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44468.log.gz 2024-03-28T15:11:22.013 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.51969.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-28T15:11:22.013 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47242.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51969.log.gz 2024-03-28T15:11:22.013 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49761.log 2024-03-28T15:11:22.013 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47242.log.gz 2024-03-28T15:11:22.013 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45142.log 2024-03-28T15:11:22.013 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52324.log 2024-03-28T15:11:22.013 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45142.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49761.log.gz 2024-03-28T15:11:22.013 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47485.log 2024-03-28T15:11:22.013 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45142.log.gz 2024-03-28T15:11:22.014 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54791.log 2024-03-28T15:11:22.014 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-28T15:11:22.014 INFO:teuthology.orchestra.run.smithi186.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52324.log.gz 2024-03-28T15:11:22.014 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45028.log 2024-03-28T15:11:22.014 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47485.log.gz 2024-03-28T15:11:22.014 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47990.log 2024-03-28T15:11:22.014 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54791.log.gz/var/log/ceph/ceph-client.admin.45028.log: 2024-03-28T15:11:22.014 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46696.log 2024-03-28T15:11:22.015 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47990.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45028.log.gz 2024-03-28T15:11:22.015 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49320.log 2024-03-28T15:11:22.015 INFO:teuthology.orchestra.run.smithi186.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47990.log.gz 2024-03-28T15:11:22.015 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.46696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37447.log 2024-03-28T15:11:22.015 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49320.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.46696.log.gz --verbose 2024-03-28T15:11:22.015 INFO:teuthology.orchestra.run.smithi186.stderr: -- /var/log/ceph/ceph-client.admin.52282.log 2024-03-28T15:11:22.015 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.37447.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49320.log.gz 2024-03-28T15:11:22.015 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56477.log 2024-03-28T15:11:22.015 INFO:teuthology.orchestra.run.smithi186.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37447.log.gz 2024-03-28T15:11:22.016 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45722.log 2024-03-28T15:11:22.016 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.56477.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.52282.log.gz -5 2024-03-28T15:11:22.016 INFO:teuthology.orchestra.run.smithi186.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44470.log 2024-03-28T15:11:22.016 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45722.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56477.log.gz 2024-03-28T15:11:22.016 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53315.log 2024-03-28T15:11:22.016 INFO:teuthology.orchestra.run.smithi186.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45722.log.gz 2024-03-28T15:11:22.016 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43618.log 2024-03-28T15:11:22.016 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53315.log: gzip 88.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.44470.log.gz -- 2024-03-28T15:11:22.016 INFO:teuthology.orchestra.run.smithi186.stderr: /var/log/ceph/ceph-client.admin.43832.log 2024-03-28T15:11:22.017 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43618.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53315.log.gz 2024-03-28T15:11:22.017 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55957.log 2024-03-28T15:11:22.017 INFO:teuthology.orchestra.run.smithi186.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43618.log.gz 2024-03-28T15:11:22.017 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44000.log 2024-03-28T15:11:22.017 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55957.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43832.log.gz 2024-03-28T15:11:22.017 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49701.log 2024-03-28T15:11:22.017 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44000.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55957.log.gz 2024-03-28T15:11:22.017 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55789.log 2024-03-28T15:11:22.017 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44000.log.gz 2024-03-28T15:11:22.018 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45974.log 2024-03-28T15:11:22.018 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49701.log.gz 2024-03-28T15:11:22.018 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45143.log 2024-03-28T15:11:22.018 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45974.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55789.log.gz 2024-03-28T15:11:22.018 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47200.log 2024-03-28T15:11:22.018 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53358.log 2024-03-28T15:11:22.018 INFO:teuthology.orchestra.run.smithi186.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45974.log.gz 2024-03-28T15:11:22.018 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47200.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45143.log.gz 2024-03-28T15:11:22.019 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55873.log 2024-03-28T15:11:22.019 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47200.log.gz 2024-03-28T15:11:22.019 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45512.log 2024-03-28T15:11:22.019 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53358.log: /var/log/ceph/ceph-client.admin.55873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44474.log 2024-03-28T15:11:22.019 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53358.log.gz 2024-03-28T15:11:22.019 INFO:teuthology.orchestra.run.smithi186.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55873.log.gz 2024-03-28T15:11:22.019 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45428.log 2024-03-28T15:11:22.019 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44474.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45512.log.gz 2024-03-28T15:11:22.020 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51884.log 2024-03-28T15:11:22.020 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45428.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44474.log.gz 2024-03-28T15:11:22.020 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-28T15:11:22.020 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45428.log.gz 2024-03-28T15:11:22.020 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.51884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52493.log 2024-03-28T15:11:22.020 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.1.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51884.log.gz 2024-03-28T15:11:22.020 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52409.log 2024-03-28T15:11:22.020 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44471.log 2024-03-28T15:11:22.020 INFO:teuthology.orchestra.run.smithi186.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52493.log.gz 2024-03-28T15:11:22.021 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55059.log 2024-03-28T15:11:22.021 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52409.log.gz/var/log/ceph/ceph-client.admin.44471.log: 2024-03-28T15:11:22.021 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46497.log 2024-03-28T15:11:22.021 INFO:teuthology.orchestra.run.smithi186.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44471.log.gz 2024-03-28T15:11:22.021 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54471.log 2024-03-28T15:11:22.021 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55059.log.gz 2024-03-28T15:11:22.021 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.0.46497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53442.log 2024-03-28T15:11:22.021 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54875.log 2024-03-28T15:11:22.021 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54471.log.gz 2024-03-28T15:11:22.022 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53526.log 2024-03-28T15:11:22.022 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54875.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53442.log.gz 2024-03-28T15:11:22.022 INFO:teuthology.orchestra.run.smithi186.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54513.log 2024-03-28T15:11:22.022 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54875.log.gz 2024-03-28T15:11:22.022 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45554.log 2024-03-28T15:11:22.022 INFO:teuthology.orchestra.run.smithi186.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53526.log.gz 2024-03-28T15:11:22.022 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54429.log 2024-03-28T15:11:22.022 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54513.log.gz 2024-03-28T15:11:22.022 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53231.log 2024-03-28T15:11:22.023 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56267.log 2024-03-28T15:11:22.023 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54429.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45554.log.gz 2024-03-28T15:11:22.023 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44810.log 2024-03-28T15:11:22.023 INFO:teuthology.orchestra.run.smithi186.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54429.log.gz 2024-03-28T15:11:22.023 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53231.log.gz 2024-03-28T15:11:22.023 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.56267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56185.log 2024-03-28T15:11:22.023 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48032.log 2024-03-28T15:11:22.023 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44810.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56267.log.gz 2024-03-28T15:11:22.023 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.56185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54175.log 2024-03-28T15:11:22.024 INFO:teuthology.orchestra.run.smithi186.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44810.log.gz 2024-03-28T15:11:22.024 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.48032.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56185.log.gz 2024-03-28T15:11:22.024 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49616.log 2024-03-28T15:11:22.024 INFO:teuthology.orchestra.run.smithi186.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48032.log.gz 2024-03-28T15:11:22.024 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44126.log 2024-03-28T15:11:22.024 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49616.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.54175.log.gz -5 2024-03-28T15:11:22.024 INFO:teuthology.orchestra.run.smithi186.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55101.log 2024-03-28T15:11:22.024 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49404.log 2024-03-28T15:11:22.024 INFO:teuthology.orchestra.run.smithi186.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49616.log.gz 2024-03-28T15:11:22.025 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55101.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44126.log.gz 2024-03-28T15:11:22.025 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49488.log 2024-03-28T15:11:22.025 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55101.log.gz 2024-03-28T15:11:22.025 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49404.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31162.log 2024-03-28T15:11:22.025 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52788.log 2024-03-28T15:11:22.025 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49488.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49404.log.gz 2024-03-28T15:11:22.025 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.tmp-client.admin.31162.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31162.log.gz 2024-03-28T15:11:22.025 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52113.log 2024-03-28T15:11:22.025 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49488.log.gz 2024-03-28T15:11:22.026 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46990.log 2024-03-28T15:11:22.026 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52113.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52788.log.gz 2024-03-28T15:11:22.026 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45141.log 2024-03-28T15:11:22.026 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52113.log.gz 2024-03-28T15:11:22.026 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.46990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54555.log 2024-03-28T15:11:22.026 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47158.log 2024-03-28T15:11:22.026 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45141.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46990.log.gz 2024-03-28T15:11:22.026 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52240.log 2024-03-28T15:11:22.026 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45141.log.gz 2024-03-28T15:11:22.027 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47158.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54555.log.gz 2024-03-28T15:11:22.027 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56125.log 2024-03-28T15:11:22.027 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47158.log.gz 2024-03-28T15:11:22.027 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49971.log 2024-03-28T15:11:22.027 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53568.log 2024-03-28T15:11:22.027 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52240.log.gz 2024-03-28T15:11:22.027 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.56125.log: /var/log/ceph/ceph-client.admin.49971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44808.log 2024-03-28T15:11:22.027 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56125.log.gz 2024-03-28T15:11:22.027 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53568.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49971.log.gz 2024-03-28T15:11:22.028 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53942.log 2024-03-28T15:11:22.028 INFO:teuthology.orchestra.run.smithi186.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53568.log.gz 2024-03-28T15:11:22.028 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55915.log 2024-03-28T15:11:22.028 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44808.log.gz/var/log/ceph/ceph-client.admin.53942.log: 2024-03-28T15:11:22.028 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55327.log 2024-03-28T15:11:22.028 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53942.log.gz 2024-03-28T15:11:22.028 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43490.log 2024-03-28T15:11:22.028 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55327.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55915.log.gz 2024-03-28T15:11:22.028 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44807.log 2024-03-28T15:11:22.029 INFO:teuthology.orchestra.run.smithi186.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55327.log.gz 2024-03-28T15:11:22.029 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43490.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-28T15:11:22.029 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48074.log 2024-03-28T15:11:22.029 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43490.log.gz 2024-03-28T15:11:22.029 INFO:teuthology.orchestra.run.smithi186.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44807.log.gz 2024-03-28T15:11:22.029 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46654.log 2024-03-28T15:11:22.029 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47527.log 2024-03-28T15:11:22.029 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.48074.log: /var/log/ceph/ceph-client.admin.46654.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48074.log.gz 2024-03-28T15:11:22.030 INFO:teuthology.orchestra.run.smithi186.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45386.log 2024-03-28T15:11:22.030 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47527.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46654.log.gz 2024-03-28T15:11:22.030 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43448.log 2024-03-28T15:11:22.030 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47527.log.gz 2024-03-28T15:11:22.030 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53731.log 2024-03-28T15:11:22.030 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45386.log: /var/log/ceph/ceph-client.admin.43448.log: gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45386.log.gz 2024-03-28T15:11:22.030 INFO:teuthology.orchestra.run.smithi186.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43215.log 2024-03-28T15:11:22.030 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53731.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43448.log.gz 2024-03-28T15:11:22.031 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45764.log 2024-03-28T15:11:22.031 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53731.log.gz 2024-03-28T15:11:22.031 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54749.log 2024-03-28T15:11:22.031 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45764.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43215.log.gz 2024-03-28T15:11:22.031 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45144.log 2024-03-28T15:11:22.031 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45764.log.gz 2024-03-28T15:11:22.031 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56393.log 2024-03-28T15:11:22.031 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54749.log: /var/log/ceph/ceph-client.admin.45144.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54749.log.gz 2024-03-28T15:11:22.031 INFO:teuthology.orchestra.run.smithi186.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45344.log 2024-03-28T15:11:22.031 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.56393.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45144.log.gz 2024-03-28T15:11:22.032 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56393.log.gz 2024-03-28T15:11:22.032 INFO:teuthology.orchestra.run.smithi186.stderr: /var/log/ceph/ceph-client.admin.53044.log 2024-03-28T15:11:22.032 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44802.log 2024-03-28T15:11:22.032 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45344.log.gz 2024-03-28T15:11:22.032 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45932.log 2024-03-28T15:11:22.032 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44802.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53044.log.gz 2024-03-28T15:11:22.032 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47948.log 2024-03-28T15:11:22.032 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44802.log.gz 2024-03-28T15:11:22.032 INFO:teuthology.orchestra.run.smithi186.stderr:gzip/var/log/ceph/ceph-client.admin.45932.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53689.log 2024-03-28T15:11:22.033 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47948.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45932.log.gz 2024-03-28T15:11:22.033 INFO:teuthology.orchestra.run.smithi186.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47948.log.gz 2024-03-28T15:11:22.033 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47695.log 2024-03-28T15:11:22.033 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45146.log 2024-03-28T15:11:22.033 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53689.log: /var/log/ceph/ceph-client.admin.47695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52577.log 2024-03-28T15:11:22.033 INFO:teuthology.orchestra.run.smithi186.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53689.log.gz 2024-03-28T15:11:22.033 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45146.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47695.log.gz 2024-03-28T15:11:22.033 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49887.log 2024-03-28T15:11:22.033 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45146.log.gz 2024-03-28T15:11:22.034 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54261.log 2024-03-28T15:11:22.034 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49887.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52577.log.gz 2024-03-28T15:11:22.034 INFO:teuthology.orchestra.run.smithi186.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49362.log 2024-03-28T15:11:22.034 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54261.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49887.log.gz 2024-03-28T15:11:22.034 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53086.log 2024-03-28T15:11:22.034 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54261.log.gz 2024-03-28T15:11:22.034 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47032.log 2024-03-28T15:11:22.034 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53857.log 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49362.log.gz 2024-03-28T15:11:22.034 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-28T15:11:22.035 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47032.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53086.log.gz 2024-03-28T15:11:22.035 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45890.log 2024-03-28T15:11:22.035 INFO:teuthology.orchestra.run.smithi186.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47032.log.gz 2024-03-28T15:11:22.035 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49658.log 2024-03-28T15:11:22.035 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.45890.log: gzip 87.8% -5 --verbose -- /var/log/ceph/ceph-client.admin.54131.log 2024-03-28T15:11:22.035 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53857.log.gz 2024-03-28T15:11:22.035 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8%/var/log/ceph/ceph-client.admin.49658.log: -- replaced with /var/log/ceph/ceph-client.admin.45890.log.gz 2024-03-28T15:11:22.035 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-03-28T15:11:22.036 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49658.log.gz 2024-03-28T15:11:22.036 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49845.log 2024-03-28T15:11:22.036 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5/var/log/ceph/ceph-mds.e.log: --verbose -- /var/log/ceph/ceph-client.admin.53189.log 2024-03-28T15:11:22.036 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54131.log.gz 2024-03-28T15:11:22.036 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44294.log 2024-03-28T15:11:22.036 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52071.log 2024-03-28T15:11:22.036 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49845.log.gz 2024-03-28T15:11:22.036 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53189.log: /var/log/ceph/ceph-client.admin.44294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52619.log 2024-03-28T15:11:22.036 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52071.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53189.log.gz 2024-03-28T15:11:22.037 INFO:teuthology.orchestra.run.smithi186.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44294.log.gz 2024-03-28T15:11:22.037 INFO:teuthology.orchestra.run.smithi186.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52071.log.gz 2024-03-28T15:11:22.037 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-28T15:11:22.037 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5/var/log/ceph/ceph-client.admin.52619.log: --verbose -- /var/log/ceph/ceph-client.admin.44475.log 94.0% 2024-03-28T15:11:22.037 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-03-28T15:11:22.037 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mgr.y.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52619.log.gz 2024-03-28T15:11:22.037 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49803.log 2024-03-28T15:11:22.037 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54917.log 2024-03-28T15:11:22.037 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49803.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2024-03-28T15:11:22.038 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53400.log 2024-03-28T15:11:22.038 INFO:teuthology.orchestra.run.smithi186.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49803.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49574.log 2024-03-28T15:11:22.038 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-28T15:11:22.038 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54975.log 2024-03-28T15:11:22.038 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53400.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54917.log.gz 2024-03-28T15:11:22.038 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46948.log 2024-03-28T15:11:22.038 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53400.log.gz 2024-03-28T15:11:22.038 INFO:teuthology.orchestra.run.smithi186.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49574.log.gz 2024-03-28T15:11:22.038 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54975.log: gzip 88.3% -5 --verbose -- /var/log/ceph/ceph-client.admin.55663.log 2024-03-28T15:11:22.039 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54975.log.gz 2024-03-28T15:11:22.039 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.46948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44168.log 2024-03-28T15:11:22.039 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55663.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46948.log.gz 2024-03-28T15:11:22.039 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53773.log 2024-03-28T15:11:22.039 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3%/var/log/ceph/ceph-client.admin.44168.log: -- replaced with /var/log/ceph/ceph-client.admin.55663.log.gz 2024-03-28T15:11:22.039 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55537.log 2024-03-28T15:11:22.039 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44168.log.gz 2024-03-28T15:11:22.039 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45027.log 2024-03-28T15:11:22.039 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53773.log: /var/log/ceph/ceph-client.admin.55537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43123.log 2024-03-28T15:11:22.040 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8%/var/log/ceph/ceph-client.admin.45027.log: -- replaced with /var/log/ceph/ceph-client.admin.53773.log.gz 2024-03-28T15:11:22.040 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55537.log.gz 2024-03-28T15:11:22.040 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45470.log 2024-03-28T15:11:22.040 INFO:teuthology.orchestra.run.smithi186.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45027.log.gz 2024-03-28T15:11:22.040 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52197.log 2024-03-28T15:11:22.040 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43123.log: /var/log/ceph/ceph-client.admin.45470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47284.log 2024-03-28T15:11:22.040 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52197.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43123.log.gz 2024-03-28T15:11:22.040 INFO:teuthology.orchestra.run.smithi186.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45470.log.gz 2024-03-28T15:11:22.041 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-28T15:11:22.041 INFO:teuthology.orchestra.run.smithi186.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52197.log.gz 2024-03-28T15:11:22.041 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48220.log 2024-03-28T15:11:22.041 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47284.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47368.log 2024-03-28T15:11:22.041 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.48220.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47284.log.gz 2024-03-28T15:11:22.041 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45148.log 2024-03-28T15:11:22.041 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48220.log.gz 2024-03-28T15:11:22.041 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43958.log 2024-03-28T15:11:22.041 INFO:teuthology.orchestra.run.smithi186.stderr: 89.7%/var/log/ceph/ceph-client.admin.45148.log: -- replaced with /var/log/ceph/ceph-client.admin.47368.log.gz 2024-03-28T15:11:22.042 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53899.log 2024-03-28T15:11:22.042 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45148.log.gz 2024-03-28T15:11:22.042 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43958.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43958.log.gz 2024-03-28T15:11:22.042 INFO:teuthology.orchestra.run.smithi186.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53629.log 2024-03-28T15:11:22.042 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52011.log 2024-03-28T15:11:22.042 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53899.log: /var/log/ceph/ceph-client.admin.53629.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53899.log.gz 2024-03-28T15:11:22.042 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53629.log.gz 2024-03-28T15:11:22.042 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-28T15:11:22.043 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54833.log 2024-03-28T15:11:22.043 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52011.log: /var/log/ceph/ceph.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52011.log.gz 2024-03-28T15:11:22.043 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56225.log 2024-03-28T15:11:22.043 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54833.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54833.log.gz 2024-03-28T15:11:22.043 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49278.log 2024-03-28T15:11:22.043 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.56225.log: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-28T15:11:22.043 INFO:teuthology.orchestra.run.smithi186.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56225.log.gz 2024-03-28T15:11:22.043 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-28T15:11:22.043 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44252.log 2024-03-28T15:11:22.044 INFO:teuthology.orchestra.run.smithi186.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49278.log.gz 2024-03-28T15:11:22.044 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47653.log 2024-03-28T15:11:22.044 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44252.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44252.log.gz 2024-03-28T15:11:22.044 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43790.log 2024-03-28T15:11:22.044 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47653.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47653.log.gz 2024-03-28T15:11:22.044 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-28T15:11:22.044 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43790.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43790.log.gz 2024-03-28T15:11:22.044 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43702.log 2024-03-28T15:11:22.044 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mds.c.log: 93.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-28T15:11:22.045 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54675.log 2024-03-28T15:11:22.045 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.43702.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43702.log.gz 2024-03-28T15:11:22.045 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47116.log 2024-03-28T15:11:22.045 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54675.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54675.log.gz 2024-03-28T15:11:22.045 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47822.log 2024-03-28T15:11:22.045 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47116.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47116.log.gz 2024-03-28T15:11:22.045 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44084.log 2024-03-28T15:11:22.045 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47822.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47822.log.gz 2024-03-28T15:11:22.045 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47611.log 2024-03-28T15:11:22.046 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44084.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44084.log.gz 2024-03-28T15:11:22.046 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50055.log 2024-03-28T15:11:22.046 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47611.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47611.log.gz 2024-03-28T15:11:22.046 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55705.log 2024-03-28T15:11:22.046 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.50055.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50055.log.gz 2024-03-28T15:11:22.046 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55411.log 2024-03-28T15:11:22.046 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55705.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55705.log.gz 2024-03-28T15:11:22.046 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44467.log 2024-03-28T15:11:22.047 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55411.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55411.log.gz 2024-03-28T15:11:22.047 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53484.log 2024-03-28T15:11:22.047 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.44467.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44467.log.gz 2024-03-28T15:11:22.047 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48178.log 2024-03-28T15:11:22.047 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53484.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53484.log.gz 2024-03-28T15:11:22.047 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47864.log 2024-03-28T15:11:22.047 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.48178.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48178.log.gz 2024-03-28T15:11:22.047 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53815.log 2024-03-28T15:11:22.047 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47864.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47864.log.gz 2024-03-28T15:11:22.048 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55143.log 2024-03-28T15:11:22.048 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53815.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53815.log.gz 2024-03-28T15:11:22.048 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52679.log 2024-03-28T15:11:22.048 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55143.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55143.log.gz 2024-03-28T15:11:22.048 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47737.log 2024-03-28T15:11:22.048 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52679.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52679.log.gz 2024-03-28T15:11:22.048 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53146.log 2024-03-28T15:11:22.048 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47737.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47737.log.gz 2024-03-28T15:11:22.049 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55495.log 2024-03-28T15:11:22.049 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.53146.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53146.log.gz 2024-03-28T15:11:22.049 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46864.log 2024-03-28T15:11:22.049 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55495.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55495.log.gz 2024-03-28T15:11:22.049 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52876.log 2024-03-28T15:11:22.049 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.46864.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46864.log.gz 2024-03-28T15:11:22.049 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49532.log 2024-03-28T15:11:22.049 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52876.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52876.log.gz 2024-03-28T15:11:22.049 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47326.log 2024-03-28T15:11:22.049 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49532.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49532.log.gz 2024-03-28T15:11:22.050 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52830.log 2024-03-28T15:11:22.050 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47326.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47326.log.gz 2024-03-28T15:11:22.050 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54303.log 2024-03-28T15:11:22.050 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.52830.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52830.log.gz 2024-03-28T15:11:22.050 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47906.log 2024-03-28T15:11:22.050 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54303.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54303.log.gz 2024-03-28T15:11:22.050 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54615.log 2024-03-28T15:11:22.053 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.47906.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47906.log.gz 2024-03-28T15:11:22.053 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51927.log 2024-03-28T15:11:22.053 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.54615.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54615.log.gz 2024-03-28T15:11:22.053 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56309.log 2024-03-28T15:11:22.054 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.51927.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51927.log.gz 2024-03-28T15:11:22.054 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-28T15:11:22.054 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.56309.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56309.log.gz 2024-03-28T15:11:22.054 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55621.log 2024-03-28T15:11:22.054 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mds.a.log: 94.1% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-28T15:11:22.054 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49446.log 2024-03-28T15:11:22.054 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.55621.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55621.log.gz 2024-03-28T15:11:22.054 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.49446.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49446.log.gz 2024-03-28T15:11:22.062 INFO:teuthology.orchestra.run.smithi186.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-28T15:11:22.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mon.c.log: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-28T15:11:22.176 INFO:teuthology.orchestra.run.smithi191.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-28T15:11:22.460 INFO:teuthology.orchestra.run.smithi186.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-28T15:11:23.415 INFO:teuthology.orchestra.run.smithi186.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-28T15:11:23.786 INFO:teuthology.orchestra.run.smithi186.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-28T15:11:23.929 INFO:teuthology.orchestra.run.smithi191.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-28T15:11:24.504 INFO:teuthology.orchestra.run.smithi186.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-28T15:11:24.528 INFO:teuthology.orchestra.run.smithi191.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-28T15:11:24.599 INFO:teuthology.orchestra.run.smithi191.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-28T15:11:25.385 INFO:teuthology.orchestra.run.smithi186.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-28T15:11:25.968 INFO:teuthology.orchestra.run.smithi191.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-28T15:11:34.102 INFO:teuthology.orchestra.run.smithi191.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-28T15:11:52.305 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-03-28T15:11:52.307 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-28T15:11:52.307 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m30.395s 2024-03-28T15:11:52.307 INFO:teuthology.orchestra.run.smithi191.stderr:user 0m52.546s 2024-03-28T15:11:52.307 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m1.722s 2024-03-28T15:11:59.185 INFO:teuthology.orchestra.run.smithi186.stderr: 95.6% -- replaced with /var/log/ceph/ceph-client.0.46497.log.gz 2024-03-28T15:11:59.187 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-28T15:11:59.187 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m37.306s 2024-03-28T15:11:59.187 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m45.588s 2024-03-28T15:11:59.187 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m1.842s 2024-03-28T15:11:59.187 INFO:tasks.ceph:Archiving logs... 2024-03-28T15:11:59.188 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984/remote/smithi186/log 2024-03-28T15:11:59.189 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T15:12:05.294 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984/remote/smithi191/log 2024-03-28T15:12:05.295 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T15:12:12.302 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T15:12:12.314 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T15:12:12.314 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T15:12:12.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T15:12:12.400 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T15:12:12.409 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T15:12:12.410 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T15:12:12.412 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T15:12:12.426 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-03-28T15:12:12.428 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-28T15:12:12.431 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T15:12:12.431 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-03-28T15:12:12.431 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 +776us[ +738us] +/- 71ms 2024-03-28T15:12:12.431 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -528us[ -566us] +/- 49ms 2024-03-28T15:12:12.431 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -60us[ -98us] +/- 79ms 2024-03-28T15:12:12.432 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:12:12.433 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T15:12:12.433 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-28T15:12:12.433 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 +762us[ +757us] +/- 71ms 2024-03-28T15:12:12.434 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -533us[ -538us] +/- 49ms 2024-03-28T15:12:12.434 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 -76us[ -81us] +/- 79ms 2024-03-28T15:12:12.434 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:12:12.434 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T15:12:12.444 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T15:12:12.445 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T15:12:12.469 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T15:12:12.549 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T15:12:12.620 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T15:12:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:12:12.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T15:12:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:12:12.683 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T15:12:12.694 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T15:12:12.736 INFO:teuthology.task.internal:Duration was 1096.370300 seconds 2024-03-28T15:12:12.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T15:12:12.777 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T15:12:12.778 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T15:12:12.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T15:12:12.823 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T15:12:12.824 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T15:12:13.104 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T15:12:13.104 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-03-28T15:12:13.104 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T15:12:13.124 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-03-28T15:12:13.125 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T15:12:13.152 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T15:12:13.152 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T15:12:13.169 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T15:12:13.222 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T15:12:13.222 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T15:12:13.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T15:12:13.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T15:12:13.377 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T15:12:13.377 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T15:12:13.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T15:12:13.433 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T15:12:13.445 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T15:12:13.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T15:12:13.471 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-03-28T15:12:13.498 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-03-28T15:12:13.519 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T15:12:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:12:13.542 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T15:12:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:12:13.573 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T15:12:13.589 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T15:12:13.590 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984/remote/smithi186 2024-03-28T15:12:13.590 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T15:12:13.628 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984/remote/smithi191 2024-03-28T15:12:13.628 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T15:12:13.664 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T15:12:13.664 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T15:12:13.666 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T15:12:13.713 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T15:12:13.725 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T15:12:13.726 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T15:12:13.753 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T15:12:13.754 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T15:12:13.756 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T15:12:13.769 INFO:teuthology.orchestra.run.smithi186.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 15:12 /home/ubuntu/cephtest 2024-03-28T15:12:13.771 INFO:teuthology.orchestra.run.smithi191.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 15:12 /home/ubuntu/cephtest 2024-03-28T15:12:13.772 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T15:12:13.814 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T15:12:13.887 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T15:12:13.910 DEBUG:teuthology.nuke:shortname: smithi186 2024-03-28T15:12:13.910 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T15:12:13.935 DEBUG:teuthology.nuke:shortname: smithi191 2024-03-28T15:12:13.935 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T15:12:13.939 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 14:44:49.320815', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0PFIEkLyGui6wXmHsGy8PaMcdiY15HyvBvZeH/tU5plzXHweD4TBjDdVh/wR+38K6IBCIsbrxOSnb2KhpgaI4='} 2024-03-28T15:12:13.964 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627984', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 14:44:49.322392', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPl/NaYLYsV2jUAwfErgcSHMcwl9tx+xvwOnB5XTNmoI3kMfLwxGPgh4IyYfUkr9ouQzNjpKWUKCBhM0MBiE2I='} 2024-03-28T15:12:13.997 INFO:teuthology.orchestra.console.smithi186:Power off 2024-03-28T15:12:13.998 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T15:12:14.022 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-03-28T15:12:14.023 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T15:12:14.045 INFO:teuthology.orchestra.console.smithi191:Power off 2024-03-28T15:12:14.046 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T15:12:14.057 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-03-28T15:12:14.070 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2024-03-28T15:12:14.070 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T15:12:14.096 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-28T15:12:18.058 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T15:12:18.097 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T15:12:18.169 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-03-28T15:12:18.207 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-28T15:12:22.170 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T15:12:22.208 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T15:12:22.285 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-03-28T15:12:22.320 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-28T15:12:22.387 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-03-28T15:12:26.322 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T15:12:26.434 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2024-03-28T15:12:26.535 INFO:teuthology.orchestra.console.smithi191:Power off completed 2024-03-28T15:12:26.641 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 1096.3703000545502 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi186 with status 128: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=49afb54454524bd3a776f2646495f48c status: fail success: false 2024-03-28T15:12:26.641 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T15:12:26.733 INFO:teuthology.run:FAIL